g.p. anders

What's New in Neovim 0.11

Wed, 26 Mar 2025 00:00:00 +0000

Neovim 0.11 was just released. As in previous installments in this series, let’s talk a bit about some of the big highlights! As always, the full list of changes can be found in the release notes (use :h news to read inside of Neovim).

Breaking Changes

We make a concerted effort to avoid breaking changes. We want the upgrade path between versions to be smooth and simple, but occasionally breaking changes are necessary (often times, a breaking change is a bug fix, or there is no way to introduce a transition path between old and new behavior). The full list of breaking changes can be found in the release notes (:h news-breaking). Plugin authors are especially encouraged to carefully read this section, as many of the changes have to do with “lower level” APIs.

LSP

Simpler LSP setup and configuration

Since the LSP client was first introduced to Neovim in version 0.5 the nvim-lspconfig plugin has been a near requirement. This caused some confusion and introduced additional complexity. Neovim supposedly had LSP “builtin”, so why was a separate plugin necessary? And for new users, the process of installing a plugin is complex and unclear. Neovim does not (yet) ship with a builtin plugin manager, so newcomers to Neovim were faced with the task of choosing and installing one of dozens of different plugin managers just to access some of Neovim’s builtin features.

For a long time this situation has been deemed unsatisfactory, both by users (we’re well aware of the “Neovim is too hard to setup” memes!) and by many on the core team (with first attempts to improve the situation happening almost 3 years ago). The difficulty for the core team was deciding how exactly to improve the situation. How could we make onboarding to Neovim and using LSP simpler and easier, without introducing too much “magic” or being too opinionated? Neovim is unapologetically a tool for power users, so any efforts to make things easier for new users should not come at the cost of (overly) inconveniencing experienced users.

Discussions around these topics occurred many, many times over several years, often ending in disagreement. The situation remained unresolved because we could not agree on a design that satisfied everyone.

Finally, late in 2024, Lewis Russell (@lewis6991 on GitHub) proposed two new interfaces: vim.lsp.config() and vim.lsp.enable(). These are high-level user facing APIs designed with the explicit goal of eventually obviating most of the code in nvim-lspconfig, turning it into a simple “bag of configs”.

With these new APIs, configuring LSP in Neovim becomes quite a bit simpler and makes it much easier to use LSP without nvim-lspconfig. For example, to configure clangd, a user can use the following:

vim.lsp.config.clangd = {
  cmd = { 'clangd', '--background-index' },
  root_markers = { 'compile_commands.json', 'compile_flags.txt' },
  filetypes = { 'c', 'cpp' },
}

vim.lsp.enable({'clangd'})

The user must provide a few required options for each server (the command to execute, the “root markers” used to determine the root directory of the workspace, and the filetypes associated with this LSP server). The explicit enable call is used to decouple configuration from enabling: this allows plugins to provide configurations for a server while still allowing the user to determine when and if a given configuration should be enabled.

However, another great aspect of this feature is that Neovim will also scan files on the user’s runtimepath for LSP configurations. The same clangd example above can instead be achieved by creating a file ~/.config/nvim/lsp/clangd.lua with the contents¹:

return {
  cmd = { 'clangd', '--background-index' },
  root_markers = { 'compile_commands.json', 'compile_flags.txt' },
  filetypes = { 'c', 'cpp' },
}

Then you only need to call vim.lsp.enable() with all of the LSP servers you want to auto-enable. The name of the server corresponds to the name of the file under the lsp/ directory (e.g. if you have lsp/clangd.lua, lsp/gopls.lua, and lsp/rust-analyzer.lua, you can add vim.lsp.enable({'clangd', 'gopls', 'rust-analyzer'}) to your init.lua file).

This follows the model used by Vim and Neovim for other configuration files, such as filetype plugins (ftplugins), color schemes, and Lua modules: place the file in your runtimepath and let Neovim discover it. The goal is to eventually have nvim-lspconfig be just a bundle of simple config files under an lsp/ directory to provide some convenient out of the box configurations.

Use :checkhealth lsp to view LSP related diagnostics and to view all of your configured LSP servers.

Builtin auto-completion

Another big feature coming to LSP in this release is builtin auto-completion. Neovim has long had manual completion (inherited from Vim) and in the prior release, we set the default omnifunc to use LSP completion sources, so users could manually trigger LSP completion using <C-X><C-O>. In this release we take this a step further by providing opt-in auto-completion from LSP sources. Credit goes to Mathias Fußenegger (@mfussenegger on GitHub) for writing the original implementation in his nvim-lsp-compl plugin and to Maria José Solano (@MariaSolOs) for porting to Neovim (and for various other improvements).

Auto-completion can be enabled using the vim.lsp.completion.enable() function. Example:

vim.api.nvim_create_autocmd('LspAttach', {
  callback = function(ev)
    local client = vim.lsp.get_client_by_id(ev.data.client_id)
    if client:supports_method('textDocument/completion') then
      vim.lsp.completion.enable(true, client.id, ev.buf, { autotrigger = true })
    end
  end,
})

Improved hover documentation

The vim.lsp.buf.hover() function (mapped to K in Normal mode by default) shows a floating window displaying documentation of the symbol under the cursor. This window now uses Markdown tree-sitter highlighting, which has a number of benefits, including highlighting code regions for other languages (using tree-sitter injections), so long as the parser for that language is installed. Concretely, this means that if the documentation for a function contains a code sample, the code sample will be properly syntax highlighted.

Users may find that the borders on their hover windows have vanished after updating. This is because Neovim no longer uses global callbacks for LSP responses (a necessary breaking change to correctly support multiple LSP clients in a buffer), so overriding vim.lsp.handlers['textDocument/hover'] to add borders to the hover window will no longer work. Fortunately, there is also a new option in this release called winborder which sets the default border for all floating windows. For example, use vim.o.winborder = 'rounded' to use rounded borders on all floating windows.

Putting it all together

The video below demonstrates how simple it is to set up LSP completely from scratch: no extra plugins required.

See :h lsp-quickstart for details.

Tree-sitter performance

Many improvements have been made to improve tree-sitter performance. Special thanks to @ribru17 who was the main driving force for many of these improvements!

Tree-sitter highlighting, folding, and injected query iteration are asynchronous. This dramatically improves startup times for large files (#31631, #31827, #32000). To quote one reviewer:

Just tried this locally, wow, this kicks ass.
Changes to how tree-sitter queries are cached reduces the need to re-parse queries, which improves performance significantly for large queries (#31974)

Better emoji support

Incorrect display of emojis which use ZWJ or other modifiers was another long standing issue (open since 2017). This and other Unicode issues were resolved in Neovim 0.11. Grapheme clusters should now display appropriately and have the correct width.

Neovim also supports DEC mode 2027 to indicate to the parent terminal that it supports proper grapheme clustering (rather than using legacy methods like wcwidth). For more info, see the Terminal Unicode Core Specification and Mitchell Hashimoto’s blog post on Grapheme Clusters and Terminal Emulators.

Diagnostics

Virtual text handler changed from opt-out to opt-in

Neovim no longer enables the “virtual text” diagnostic handler by default. It must now be opted-into explicitly with

vim.diagnostic.config({ virtual_text = true })

The virtual text handler also has a new current_line option to only enable virtual text diagnostics for the current cursor line:

vim.diagnostic.config({
  virtual_text = { current_line = true }
})

Virtual lines

GitHub user @WhyNotHugo created a plugin called lsp_lines.nvim which shows diagnostics as separate “virtual lines” in a buffer. We reached out to Hugo a while ago to ask if he’d be interested in collaborating to upstream this plugin to Neovim. He was supportive, but the effort stalled out. That is, until about 2 months ago when Maria José Solano (@MariaSolOs) picked up the torch and put together a PR adding this feature into Neovim core. Many thanks to Hugo for supporting this effort and for creating the original plugin in the first place, and to Maria for doing the work of porting!

New virtual lines diagnostic handler @WhyNotHugo

To enable this feature, use

vim.diagnostic.config({
  -- Use the default configuration
  virtual_lines = true

  -- Alternatively, customize specific options
  -- virtual_lines = {
  --  -- Only show virtual line diagnostics for the current cursor line
  --  current_line = true,
  -- },
})

Defaults

More default mappings

In addition to the new configuration APIs detailed above, we’ve also tried to make LSP simpler for new users by adding some more default key mappings. These mappings will not override users existing mappings, so experienced users are free to ignore them and use whatever mappings they are used to. The goal is to eliminate some steps for brand new users.

grn in Normal mode maps to vim.lsp.buf.rename()
grr in Normal mode maps to vim.lsp.buf.references()
gri in Normal mode maps to vim.lsp.buf.implementation()
gO in Normal mode maps to vim.lsp.buf.document_symbol() (this is analogous to the gO mappings in help buffers and :Man page buffers to show a “table of contents”)
gra in Normal and Visual mode maps to vim.lsp.buf.code_action()
CTRL-S in Insert and Select mode maps to vim.lsp.buf.signature_help()
[d and ]d move between diagnostics in the current buffer ([D jumps to the first diagnostic, ]D jumps to the last)

We’ve also included versions of some of the mappings from Tim Pope’s vim-unimpaired:

[q, ]q, [Q, ]Q, [CTRL-Q, ]CTRL-Q navigate through the quickfix list
[l, ]l, [L, ]L, [CTRL-L, ]CTRL-L navigate through the location list
[t, ]t, [T, ]T, [CTRL-T, ]CTRL-T navigate through the tag matchlist
[a, ]a, [A, ]A navigate through the argument list
[b, ]b, [B, ]B navigate through the buffer list
[<Space>, ]<Space> add an empty line above and below the cursor

Terminal

The embedded terminal emulator has received a lot of love during this release cycle. Some highlights:

Programs in the terminal can now change the shape and blink of the cursor (this feature request was created in 2015, almost 10 years ago!)
Programs in the terminal can use the OSC 52 escape sequence to interact with the user’s clipboard. Note: this is separate from Neovim’s own OSC 52 support, which is used for copying and pasting text in normal buffers.
Programs in the terminal can use the OSC 8 escape sequence to emit hyperlinks. Neovim will detect these and add appropriate metadata to make them clickable in the parent terminal emulator.
When a program in the terminal enables DEC mode 2031 (theme update notifications), Neovim will send a notification to the program in the terminal whenever the 'background' option changes. This is useful for automatically updating color schemes based on the lightness or darkness of the background color. This works in the main Neovim instance too. Demo
The terminal now supports a subset of the kitty keyboard protocol (“Disambiguate escape codes”). Programs that use the kitty keyboard protocol (such as Neovim itself!) will now work when running in Neovim’s terminal.
New mappings [[ and ]] will jump between shell prompts in a terminal buffer as long as the shell emits semantic prompt markers (OSC 133 sequences). Fish 4.0 emits these by default, and many terminal emulators come with “shell integration” scripts to enable these markers. This also allows you to set up prompt markers in your terminal (see :h shell-prompt-signs for example code).

Miscellaneous

Use g== in Normal mode in a help buffer to execute a block of code. This is useful for running code examples in help docs (example use case: you open up a log file that contains terminal escape sequences. Run :h terminal-scrollback-pager, move your cursor over the example code for the :TermHl command, press g==, and now run :TermHl in the log file buffer).
The completeopt option has a new “fuzzy” value that enables fuzzy completion.
Extmarks now have the ability to conceal entire lines, rather than just characters within a line. This is used, for example, in Markdown buffers to hide code fences (e.g. ```) when conceallevel is set without leaving empty lines.
New right-click menu items include “Go to definition” and “Open in web browser” (when Neovim can detect that you are right clicking on a URL).

Getting Involved

There is nothing quite as fun as hacking on your own tools. If you are a Neovim user, I encourage you to get involved, whether that is through contributing directly to the Neovim project or a plugin, or simply participating in community spaces such as the Neovim Matrix room. If you have pain points or frustrations when using Neovim, please let us know. Many of us on the core maintenance team have been using Vim/Neovim for so long that we’ve forgotten what it’s like to be a beginner and, speaking for myself at least, this can leave blind spots on how things can be improved.

Thanks for reading and for using Neovim!

More examples here. These examples use Fennel rather than Lua, but the concept is the same. ↩︎

Ghostty Is Native—So What?

Fri, 20 Dec 2024 00:00:00 +0000

Ghostty is a new terminal emulator by Mitchell Hashimoto. While a lot has been said about Ghostty’s performance, less discussed is its native platform¹ integration which is, in my opinion, its most distinctive and underrated feature.

Despite being listed as a key motivating factor in Mitchell’s original introduction, online discussions about Ghostty rarely mention its native integration, and when they do many commenters remain skeptical over its importance or don’t understand what that means. So I want to briefly describe some of the ways that Ghostty takes advantage of native OS integration so you can see for yourself what makes it stand out.

Ghostty is cross-platform, but I’m only going to talk about the macOS application here, partly because it’s what I personally use most of the time, but also because the native integration on macOS is more full featured. macOS provides a lot of features and UI affordances that applications can take advantage of to provide a coherent, integrated experience. Ghostty makes use of many of these features, making it feel more native than most other terminal emulators—iTerm2 being the main exception.

On Linux, “native” is a murkier concept because there’s not a single desktop environment. Instead, applications integrate with specific environments like GNOME or KDE through their GUI frameworks (GTK/Adwaita or Qt). Ghostty uses GTK and (optionally) Adwaita, making it most native to GNOME while remaining visually consistent in KDE—though without as much deep system integration as in macOS. I’ve noticed that many Linux users don’t care about apps being “native”, which is understandable. In my experience, this is something that macOS users tend to care about more.

If after reading this you remain unconvinced that being a native application is something that one should care about in a terminal emulator, then that’s fine! Not everyone values the same things in their software. But at least we’ll be on the same page on exactly what it is you don’t care about.

What is a native application?

Let’s get the definition out of the way. A “native” application is one which uses the operating system’s GUI toolkits, SDKs, and other libraries (e.g. font rendering) to create a user experience that is coherent and consistent with the rest of the operating system and other applications on the operating system. This does not only cover UI elements like buttons and tabs, but the entire user experience, from builtin key bindings, menu items, and other kinds of system interactions.

Native Tabs

Ghostty uses native tabs which have the same look and feel as tabs found in applications like Safari, Preview, Finder, and Terminal.app.

This also means that native features like pressing Shift+⌘+\ open the tab overview, just as in other applications.

Proxy Icon

Ghostty uses a “proxy icon” in the window title bar that gives you access to the current working directory of the terminal. You can use this like any other proxy icon in macOS: drag it into another application or right-click to interact with it.

Look Up

Triple tap on the trackpad to use the macOS “look up” feature to find the definition of a word or preview a URL:

Secure Keyboard Entry

macOS has a feature called secure keyboard entry which prevents any other application from detecting or recording what you type (it also prevents other applications from stealing focus, which is important when you’re entering sensitive data like passwords). Only Ghostty, iTerm2, and Terminal.app support this feature. Ghostty automatically enables it when you’re typing a password—though it can also be enabled manually. Note the icon in the upper right corner of the window:

Window Restoration

A cool feature of macOS is that application state can be serialized so that when you re-launch the application later, you can pick up right where you left off. Ghostty supports this too:

For now only the window position and tabs/splits are restored, not the actual terminal contents (like you see in Terminal.app). Restoring the terminal contents is something that’s on the long term roadmap though.

Conclusion

Hopefully these examples clarify what I and others mean when we talk about Ghostty being a “native application”. I didn’t showcase Linux here at all, but it is worth mentioning that for GNOME users especially Ghostty feels like a native app too. And for non-GNOME users, the Adwaita integration and most of the GTK windowing features can be disabled, so that Ghostty looks just as at home in a barebones tiling window manager as it does in a full-fledged desktop environment.

That’s all! Thanks for reading.

As of this writing, that is macOS and Linux ↩︎

What's New in Neovim 0.10

Thu, 16 May 2024 00:00:00 +0000

Neovim 0.10 was the longest release cycle since the heady days of the 0.5 release. There are a ton of new features in this release (as well as some breaking changes), so be sure to check the full release notes. You can view the release notes directly in Nvim with :h news. The news file includes information on new features, deprecations, and breaking changes. I especially urge plugin authors to read this file carefully.

Like last time, in this post I’ll cover a subset of the new features. As I said, it’s a big release, and I won’t write about everything, but these are some of the highlights.

Defaults

Colorscheme

Nvim has been in need of a new default colorscheme for a long time. While the default colors have some charm and nostalgia, there are serious accessibility and aesthetic issues, particular when using Nvim is a diff viewer. The screenshot below shows the default colorscheme before Nvim 0.10:

Nvim 0.10 now ships with a new default colorscheme, which you can see below:

The new default colorscheme was heroically contributed by Evgeni Chasnovski, who endured seemingly endless bikeshedding and successfully navigated the treacherous waters of such a subjective change. There were many constraints imposed on the design of the colorscheme, which you can read in the PR (and the many follow up PRs and issues linked therein). It is quite literally impossible to please everyone with a single colorscheme, and there is no expectation that we have achieved that, but the new colorscheme does, hopefully, improve the default experience for many people.

LSP and Diagnostics Mappings

Nvim 0.10 adds a new default mapping: K in Normal mode maps to vim.lsp.buf.hover() whenever a buffer attaches to an LSP client, unless 'keywordprg' is already set to a non-default value. Use this to view information about the function or variable under the cursor. This in addition to the existing defaults mentioned in :h lsp-defaults which were added in the previous release.

In addition, there are new default mappings for navigating diagnostics:

[d and ]d in Normal mode map to vim.diagnostic.goto_prev() and vim.diagnostic.goto_next(), respectively. Use these to navigate between diagnostics in the current buffer.
<C-W>d (and <C-W><C-D>) in Normal mode map to vim.diagnostic.open_float(). Use this to view information about any diagnostics under the cursor in a floating window.

The diagnostics mappings override builtin mappings. These builtin mappings are not often used, but if you do use them you can delete the new default mappings with vim.keymap.del() or :unmap.

The intention behind providing more default mappings is to make it easier to get started with LSP in Neovim. The hope is that most users should be able to get up and running with useful LSP features with little to no configuration and without needing to write a bunch of Lua code. We have plans to add more defaults in future releases, though as you might imagine this is a fraught exercise. It is quite difficult to change default behavior in a tool as extensible and customizable as Nvim without breaking users' workflows.

Two features which many users rely on for their LSP experience are snippets support and autocompletion. Both of these features are on the roadmap to be included in Nvim core (see #27339, #25696, and #25670) which will, hopefully, make the experience of using LSP in Neovim even smoother in the future.

Builtin Commenting

Longtime Vim users are likely familiar with the venerable vim-commentary plugin, which creates mappings and operators for commenting and uncommenting text. Thanks to Evgeni Chasnovski, Nvim 0.10 now bundles this functionality by default, using a from-scratch implementation in Lua. Unlike vim-commentary, Nvim’s builtin implementation provides a text object and supports Tree-sitter, which is useful when working in injected languages (example: if you comment text inside a <script> tag in an HTML buffer, it will correctly use the Javascript // comment string rather than HTML’s  comment string).

See :h commenting for more details.

Terminal UI enhancements

Nvim 0.10 adds support for many new terminal based capabilities.

Synchronized Output

Many modern terminal emulators support synchronized output which allows terminal applications (like Nvim) to “batch” all of their UI updates in the terminal emulator and display them all at once. This can reduce “flickering” and “tearing” when a terminal UI (TUI) is being drawn very rapidly. This can sometimes happen in Nvim when using plugins that perform rapid UI updates (e.g. spinners, LSP status windows, etc.). Nvim 0.10 introduces a new 'termsync' option (enabled by default) which enables this feature if it is supported by the terminal emulator.

System clipboard synchronization

Nvim 0.10 can now use the OSC 52 escape sequence to write to (or read from) the system clipboard. OSC 52 is only used automatically when (1) it is able to determine that the terminal emulator supports it (via XTGETTCAP), (2) it is running in an SSH session, and (3) when the 'clipboard' option is unset. Note that some terminal emulators do not support reading from the system clipboard with OSC 52, or only allow it after prompting the user for confirmation. It is recommended to use the traditional “paste” key binding (e.g. Cmd+V on macOS or Ctrl+Shift+V on Linux) to paste into Nvim from the system clipboard and use "+ to copy to the clipboard inside Nvim (see :h quoteplus).

Hyperlinks

Nvim 0.10 introduces experimental support for hyperlinks using the OSC 8 sequence. By default, this is used in Markdown for links of the form [example](https://example.com). If your terminal emulator supports the OSC 8 escape sequence, then the text example in the Nvim buffer will contain a hyperlink that can be clicked and will open https://example.com in your web browser.

This means that users can view Markdown files with :set conceallevel=2 and get an experience much closer to the rendered output:

In the screenshot above, the underlined words are clickable link text, just as you would find in a web browser.

Automatic truecolor detection

Nvim 0.10 will now automatically determine if the terminal emulator supports 24 bit color (“truecolor”) and enable the 'termguicolors' option if it does. It does this through a combination of heuristics ($COLORTERM, terminfo) and terminal queries (DECRQSS, XTGETTCAP). This detection works even over SSH connections and when inside a terminal multiplexer such as tmux.

LSP inlay hints

Nvim 0.10 now supports LSP inlay hints thanks to Chinmay Dalal. A picture here will do more than my words can:

The dark colored texts which display type annotations for variable declarations are inlay hints. This text is not part of the actual source file in the buffer, but is “virtual” text inserted by Nvim and provided by the language server. These hints can be enabled or disabled dynamically using vim.lsp.inlay_hint.enable().

Many other new LSP features were added in Nvim 0.10 as well. Refer to the “LSP” section in :h news for a full list.

Tree-sitter query editor

Nvim 0.10 adds even more tools for working with Tree-sitter queries. If you didn’t know already, Nvim bundles an “inspector” which allows you to view the abstract syntax tree of any source file with the :InspectTree command (so long as a Tree-sitter parser for the file’s language is installed). Example:

Thanks to Maria José Solano, Nvim 0.10 adds a powerful capability to the Tree-sitter inspector: a query editor. The query editor allows you to write queries and see the matches in a source buffer in real time. Being able to create and modify queries interactively makes writing new queries a breeze. Example:

In the screenshot above, the query editor is open in the top-right window, the tree inspector is in the bottom-right window, and the source buffer is in the left window. Notice that the text that is matched by the query in the query editor is highlighted in the source buffer and that the text initializer (the capture group used in the query) is floating on the line next to the match. As the query is updated in the editor, the corresponding highlights in the source buffer change in real time.

The query editor can be opened by pressing o in the :InspectTree window, with the :EditQuery command, or by calling vim.treesitter.query.edit() directly.

With :InspectTree and :EditQuery, Neovim is one of the best tools (if not the best tool) for working with Tree-sitter, even if you don’t actually use Neovim.

Miscellaneous

As I mentioned at the top of this post, there are a lot of new features in this release, and I am not going to cover all of them. Here I’ll list a few more that I personally think are worth mentioning:

The :terminal command now accepts modifiers, so you can use e.g. :botright terminal (or :bo te) to open a new terminal window in a split at the bottom of your screen. In addition, :terminal buffers that are started with no arguments (and thus open a shell) will close automatically when the shell exits without an error.
gx in Normal mode calls vim.ui.open() on whatever is under the cursor, which shells out to your operating system’s “open” capability (e.g. open on macOS or xdg-open on Linux). For instance, pressing gx on a URL will open that URL in your browser.
Tree-sitter based syntax highlighting is enabled by default for Lua, Vimdoc (:help), and Tree-sitter queries (to revert to traditional regex based syntax highlighting, create a FileType autocommand or ftplugin with vim.treesitter.stop() for the respective filetype).
Q and @ in Visual mode will execute the last recorded/executed macro for all visually selected lines.
Users using a terminal emulator that supports the Kitty keyboard protocol can create mappings using the “super” and “meta” modifiers with the <D- and <T- prefixes, respectively (e.g. <D-S> is Cmd+S on macOS).

Roadmap

The Neovim project is loosely organized and structured. We follow a “fun driven development” paradigm: for the most part, contributors and maintainers work on things that are personally interesting to them. Because of this, it can be difficult to predict what will happen in future releases. If there is a feature you want to see implemented, the best way to do it is to take a crack at it yourself: many of the features mentioned in this very blog post were contributed by users that are not part of the “core” maintenance team!

There is active chatter about further improving the LSP and Tree-sitter experience (as mentioned above, builtin snippet and completion support are a work-in-progress). One of the biggest challenges with Tree-sitter is parser distribution: currently, parsers are shared object files which must be compiled on every user’s machine or distributed with a system package manager. We are already working on implementing the new Tree-sitter WASM capabilities which will enable distributing Tree-sitter parsers as portable WASM blobs. This solves the distribution problem and may enable Neovim to ship more Tree-sitter parsers out of the box.

We would also like to integrate some of the functionality from nvim-lspconfig so that, ideally, nvim-lspconfig is just a “bag of configs” without any smarts or core logic. How best to do this is still hotly debated, but it is something that is being discussed.

Getting Involved

There is nothing quite as fun as hacking on your own tools. If you are a Neovim user, I encourage you to get involved, whether that is through contributing directly to the Neovim project or a plugin, or simply participating in community spaces such as the Neovim Matrix room. If you have pain points or frustrations when using Neovim, please let us know. Many of us on the core maintenance team have been using Vim/Nvim for so long that we’ve forgotten what it’s like to be a beginner and, speaking for myself at least, this can leave blind spots on how things can be improved.

Thanks for reading and for using Neovim! Hopefully the next update will come a little sooner.

One Last Thing

The Neovim project now has a merch store! If you didn’t know that you needed a Neovim branded hoodie or mug, now you do. All proceeds raised from the store are used to fund project needs and sponsor full-time development work.

Making ijq Fast

Sat, 13 Apr 2024 13:14:07 -0500

I recently received an issue report that ijq was performing slowly. The issue claimed that, when used on a large (16 MB) JSON file, ijq was “too slow to be usable”. I downloaded the test file which was (helpfully) provided in the issue and tried it myself. Even using the most charitable definition, I agree that this classifies as “unusably slow”. Note that in this screen recording I am displaying my key presses to show just how unresponsive ijq is:

In the rest of this post I’ll talk about why ijq was performing so badly, and what I changed to fix it. The end result is that ijq is now, dare I say, not just usable, but downright snappy:

Background

First, a brief explanation of how ijq works.

ijq is, more or less, an interactive TUI wrapper around jq. When ijq starts, the input document (whether read from a file or stdin) is read and processed with jq using the default filter (.), and the output is written to the left (Input) pane. A second jq process is run which processes the input document using whatever filter the user supplied to ijq; that output is written to the right (Output) document. The input text box is the filter passed to jq. Whenever the input text box changes, jq is run on the input document using the new filter and the output is written to the right pane.

Conceptually, this is fairly simple, but there are plenty of ways to do this wrong (as I learned).

The input document is represented as a Document object:

type Document struct {
	input   string
	filter  string
	options Options
}

The Document object implements the WriterTo interface for writing to an io.Writer object:

// Filter the document with the given jq filter and options
func (d *Document) WriteTo(w io.Writer) (n int64, err error) {
	opts := d.options
	if _, ok := w.(*tview.TextView); ok {
		// Writer is a TextView, so set options accordingly
		opts.forceColor = true
		opts.monochrome = false
		opts.compact = false
		opts.rawOutput = false
	}

	args := append(opts.ToSlice(), d.filter)
	cmd := exec.Command(d.options.command, args...)
	stdin, err := cmd.StdinPipe()
	if err != nil {
		return 0, err
	}

	go func() {
		defer stdin.Close()
		_, _ = io.WriteString(stdin, d.input)
	}()

	out, err := cmd.CombinedOutput()
	if err != nil {
		if exiterr, ok := err.(*exec.ExitError); ok {
			exiterr.Stderr = out
		}
		return 0, err
	}

	if tv, ok := w.(*tview.TextView); ok {
		w = tview.ANSIWriter(tv)
		tv.Clear()
	}

	m, err := w.Write(out)
	n = int64(m)
	return n, err
}

If the Writer is a TextView (the TUI widget used for the input and output panes) then we set some options which are passed to the child jq process. The child process is started and the document’s input is piped to the process’s stdin. When the process finishes, the stdout is written to the Writer.

The filter input text box has a callback which is fired anytime its value changes:

filterInput.SetChangedFunc(func(text string) {
	go app.QueueUpdateDraw(func() {
		errorView.Clear()
		doc.filter = text
		outputView.ScrollToBeginning()
		_, err := doc.WriteTo(outputView)
		if err != nil {
			exitErr, ok := err.(*exec.ExitError)
			if ok {
				fmt.Fprint(tview.ANSIWriter(errorView), string(exitErr.Stderr))
			}

			return
		}
	})
})

Whenever the input changes, we update the current Document’s filter and call WriteTo using the output text view. If the jq subprocess returns an error, write the error text to the error text view.

These two functions are the backbone of ijq, and both have fatal flaws (experienced Go and/or UI developers have perhaps already spotted them) responsible for ijq’s slow performance.

Before moving on, it’s worth mentioning that these problems had not been noticed so far because I have only ever used ijq on relatively small inputs. When the input is small enough, the typing speed of the user becomes the bottleneck, not the jq process or updating the text views, so these issues have gone undetected (by me) for a long time (maybe other ijq users have noticed them, but until this issue report I was unaware!).

Problem

Refer back to the filter input callback implementation above. The first problem is the use of QueueUpdateDraw. ijq uses a TUI widget framework called tview to manage the application run loop. tview’s documentation for QueueUpdate states:

QueueUpdate is used to synchronize access to primitives from non-main goroutines. The provided function will be executed as part of the event loop and thus will not cause race conditions with other such update functions or the Draw() function.

QueueUpdate (and QueueUpdateDraw) is used to modify UI primitives from non-main threads. Importantly, these functions block the caller until the main thread processes the callback and synchronizes the state. tview’s wiki page on Concurrency says:

Any event handlers you install, e.g. InputField.SetChangedFunc() or Table.SetSelectedFunc(), are invoked from the main goroutine. It is safe to make changes to your primitives in these handlers. If they are invoked in response to a key event, you must not call Application.Draw() as the application’s main loop will do that for you and calling Application.Draw() (or Application.QueueUpdate() or Application.QueueUpdateDraw()) from within an event handler will lead to a deadlock.

The use of QueueUpdateDraw() in this handler is in violation of this advice. The function is started in a separate goroutine (it’s called with go), so this avoids deadlock, but it still blocks the main thread until the WriteTo function finishes. Recall that WriteTo runs the jq subprocess, so effectively this runs each jq process on the main thread every time the filter input changes.

This change was added (by me) over 3 years ago, because it (apparently) solved another issue with race conditions in the prior implementation (no doubt that it did: when everything is on the main thread there are no race conditions).

The second important problem is in the WriteTo implementation above. This function uses the CombinedOutput function to get both stdout and stderr from the jq process. This function writes the process’s stdout and stderr into a temporary buffer. After the jq process completes, that buffer is then written to the output text view.

This is slow, because we first wait for the process to write to the buffer, and only when that is finished do we start writing to the text view. This is the cause of ijq’s long startup time (visible in the first video at the top of the post). Instead, we should be able to pipe the output from the subprocess directly to the text view so that the output is visible immediately.

Solutions

Fixing the first problem was straightforward: drop QueueUpdateDraw, run WriteTo in a separate thread, and use a ChangedFunc on the output view to redraw the application when the output view changes. With this change ijq immediately becomes more responsive. User input is never blocked on the jq process.

This is suboptimal though, because a new jq process is spawned on every input change. We should be able to cancel old processes when the input changes, and if multiple input change callbacks are fired before starting a new jq process, we should only ever use the most recent input. Go’s native channels are almost a good fit here, but what we want is a LIFO (last-in, first-out), whereas Go’s channels are FIFO (first-in, first-out): every input written into a channel is read out in the same order. On the receiving end of the channel, there’s no (easy) way to only read the most recent one. So instead of channels, we use a regular old mutex and condition variable. The mutex guards changes to the Document object and a pending variable. When the filter input changes, the Document object is updated, pending is set to true, any existing jq processes are cancelled, and the condition variable is signalled. A separate thread waits for pending to become true and when it does, makes a copy of the Document object before starting a new jq process.

Fixing the second problem was a bit trickier. Go’s Cmd object can use any object that implements the io.Writer interface in its Stdout and Stderr fields. Instead of writing to a temporary buffer, waiting for the process to complete, and then copying to the text view, we can simply write to the text view immediately! This change, combined with removing another incorrect use of QueueUpdateDraw in the app startup, solves the initial startup time issue.

But sometimes fixing one issue introduces another: in this case, when the jq process writes directly to the text view, we have to clear the text view before running the jq process (or else each successive process will append to the previous process’s output). But when we do this, the output view is cleared every time the user updates the filter input. We only want the output view to change when there is actually new visible output. This ensures that if the user types in an invalid filter, the last output is still visible. This is an important property for an interactive jq tool where experimenting with the right filter is a large part of the value proposition.

To address this, we make a new type that wraps a TextView and tracks its “dirty” state. When the text view is dirty, we call the view’s Clear() method before it is written (and then reset the dirty state).

type pane struct {
	tv    *tview.TextView
	dirty bool
}

func (pane *pane) Write(p []byte) (n int, err error) {
	if pane.dirty {
		pane.tv.Clear()
		pane.dirty = false
	}

	return pane.tv.Write(p)
}

The dirty state is then set before the jq process is invoked.

With these changes, ijq’s performance problems are solved!

Conclusion

I’m not sure there is a lesson here other than re-learning that concurrency (and UIs) are hard, even in languages like Go which give you great tools to manage concurrency. Ultimately, for me, it was a learning experience and a fun excuse to hack on a side project. If you are an ijq user, I hope this makes your user experience even better!

State of the Terminal

Tue, 12 Mar 2024 00:00:00 +0000

This is a companion article to my talk at Neovimconf 2023.

I have been using Vim/Neovim as my full time text editor for close to 10 years. I’ve spent a lot of time in the terminal and have become very aware of the many flaws and idiosyncrasies of this bizarre platform. But I also think it gets a lot of things right! And I’m not alone in this belief: terminal based tools are still widely popular even in the presence of many alternatives (the StackOverflow developer survey shows that Neovim is the “most loved” editor 3 years in a row).

It’s only been in the last couple of years that I’ve begun to dig deep into the inner workings of how terminal emulators, and the applications that run inside of them, really work. I’ve learned that there is a lot of innovation and creative problem solving happening in this space, even though the underlying technology is over half a century old¹.

I’ve also found that many people who use terminal based tools (including shells like Bash and editors like Vim) know very little about terminals themselves, or some of the modern features and capabilities they can support.

In this article, we’ll discuss some of the problems that terminal based applications have historically had to deal with (and what the modern solutions are) as well as some features that modern terminal emulators support that you may not be aware of.

But first, some (very) brief history.

Background & History

Most terminal emulators today can directly trace their roots back to the DEC VT100. The VT100 was not the first video terminal, nor was it the last, but it was the most popular (at the time). And as we’ve learned from history many times since, what becomes popular creates the de facto standard for everything that comes after.

DEC VT100 Jason Scott, CC BY 2.0 via Wikimedia Commons

Video terminals were an improvement on the teletype machines that preceded them. They could move the cursor around the screen to create interactive interfaces. They could use color, and clear and redraw their displays quickly without feeding out reams of paper.

Different video terminals had their own unique way of doing things using unique, proprietary escape codes (a sequence of bytes beginning with the escape 0x1b character). This made life difficult for applications because they had to know which of these sequences to use. Libraries and helper programs (e.g. termcap) were created to help ameliorate these issues (we still live with the descendant of these early libraries, terminfo).

Eventually, formal standards were created, such as ECMA-48 and ANSI X3.64 (from which the term “ANSI escape codes” derives), which defined a set of standard escape sequences. The DEC VT100 was the first video terminal to support these new standards. Its popularity, combined with the new standards, meant that programs now had a set of known good escape sequences they could reliably use. Its popularity spawned many clones, which in turn supported the same sequences for compatibility with applications.

Graphical window systems eventually replaced hardware video terminals, but users still wanted to use the terminal based programs they were accustomed to (you know how those vi people are). In 1984, work began on a software terminal emulator at MIT. This emulator became part of the X project and was named Xterm. Xterm implemented its own features which did not exist on the video terminals it emulated, such as mouse tracking and a configurable color palette. These features were in turn copied by Xterm clones, until eventually Xterm itself became the new de facto standard.

Terminal Emulator Basics

Terminal based applications write two kinds of data to the terminal emulator: printable text that is displayed to the user, and control codes, which modify the terminal emulator’s state. Control codes are either single bytes in the C0 character set (bytes 0x00 through 0x1f) or sequences of bytes that begin with the escape character (0x1b). These sequences are most commonly referred to as “escape sequences”, and it is these sequences that do the bulk of the heavy lifting in terminal applications.

Most control codes from the C0 character set are not used today, but regardless of experience with terminals or terminal applications, most developers are likely familiar with control codes such as \r (carriage return), which moves the cursor to the beginning of the current line, and \n (line feed), which moves the cursor to the next line.

Escape sequences are varied and numerous, but the vast majority used in practice fall into one of three categories: Control Sequence Introducer (CSI), Device Control String (DCS), and Operating System Command (OSC).

CSI sequences are those which begin with the prefix ESC [ (0x1b 0x5b). Escape sequences in this category are those which reposition the cursor, change the cursor style, clear the screen, set foreground and background colors, and more.

OSC sequences are those which begin with the prefix ESC ] and are typically used for things that modify or interact with the user’s environment outside of the terminal emulator itself (hence the name “Operating System Command”). Examples are reading from or writing to the system clipboard, changing the title of the terminal emulator’s window, or sending desktop notifications.

Xterm maintains a list of all of the control sequences it supports on its website, which, along with vt100.net, forms an informal pseudo-specification for VT100 emulators. Note that this list may not contain some control sequences used by other, modern terminal emulators for features which Xterm does not support (e.g. the Kitty keyboard protocol, which we’ll discuss later).

Escape sequences are actually quite easy to use, and you can even do it straight from your shell. Try running the following command from any shell:

printf '\e[1;32mHello \e[0;4;31mworld!\n\e[0m'

This command will print the text “Hello world!”, with “Hello” in green, bold text and “world!” in red, underlined text.

The escape sequences used here are of the form CSI <parameters> m, which is so common it has its own name: Select Graphic Rendition (SGR). The SGR escape sequence sets foreground and background colors for all printed text. The first escape sequence in the example \e[1;32m enables the bold attribute (1) and sets the foreground color to green (32). The second escape sequence \e[0;4;31m first clears any existing styles (0), then enables the underline attribute (4), and finally sets the foreground text color to red (31). Finally, the last escape sequence \e[0m resets all styles back to their defaults.

Another use case for simple CSI sequences is redrawing text on the screen on an already existing line (e.g. for a progress bar or text that updates itself over time). Hint: look at \r, CSI A, and CSI K.

Most escape sequences are sent from the application to the terminal emulator, but occasionally the terminal emulator sends escape sequences to the application. Usually this is done in response to a query from the application (for instance, to determine if a certain mode is set).

Problems & Solutions

Terminal emulators are descended from old, legacy technologies, which brings with it its fair share of problems. Many of these problems have been (mostly) solved, or at least ameliorated, while others are still active areas of innovation and research.

Key Encoding

Terminal emulators and terminal applications communicate through a stream of bytes. When a user presses a key the terminal sends the byte representation of the character associated with that key. The old video terminals only supported ASCII so this was, generally, fairly straightforward.

Modifier keys like Ctrl and Alt complicate this situation. Alt modified keys are encoded by prefixing the character with an Esc. But this has a problem: including an extra Esc byte for the Alt modifier introduces ambiguity between Alt modified key presses and two separate key presses. When an application sees Esc C, should it interpret it as Alt-C or did the user press Esc and then press C? Applications usually solve this by measuring the amount of time between Esc and the next character. If the time is less than some defined interval, it is considered an Alt modified key press (Vim uses the ttimeoutlen option, tmux uses the escape-time option).

Ctrl modified keys are an even bigger problem. When Ctrl is used as a modifier, the shifted² version of the key has the 7th bit masked off (for example, C is 0x43 and after masking the 7th bit the byte becomes 0x03). This means that not only can the Shift modifier not be used in conjunction with Ctrl, but that certain Ctrl modified keys are completely indistinguishable from other control codes.

For instance, when you press the Return key the terminal emulator sends the byte \r (0x0d) to the application. But if you press Ctrl-M then the terminal emulator also sends the byte 0x0d to the application (M is 0x4d in ASCII, so when the 7th bit is masked out, it becomes 0x0d). From the application’s perspective, there is literally no way to distinguish these two events.

For a long time this meant that certain modified keys like Ctrl-I, Ctrl-J, and Ctrl-M could not be used in terminal applications like Vim. There have been a few attempts to solve this problem: the first came from Xterm in 2006 through the modifyOtherKeys option. Paul Evans (author of libvterm and libtickit) introduced an alternate key encoding using the CSI u escape sequence in an essay which is sometimes colloquially referred to as “fixterms”. The CSI u encoding proposed by Evans was extended by Kovid Goyal, the author of the kitty terminal emulator, in what has become known as the kitty keyboard protocol.

What all of these solutions have in common is that key presses are sent to the terminal application encoded as escape sequences. This eliminates any ambiguity for modified keys and enables certain modifier combinations (such as Ctrl + Shift) that are not possible using “legacy” encoding. The CSI u encoding proposed by Evans and adapted by kitty encodes a modified key press like Ctrl-M as \e[109;5u. The encoding of unmodified key presses like Return depend on which “level” of the kitty keyboard protocol is enabled. Applications can opt-in to different levels to ease adoption (for instance, Neovim uses only the first level, “Disambiguate escape keys”). See the kitty documentation for more details.

Sending key presses as escape sequences requires that terminal applications are able to recognize and parse those sequences, so it is not something that “just works” out of the box. However, the kitty keyboard protocol has been widely adopted by both modern terminal emulators and terminal applications. Terminals which support the kitty keyboard protocol (to some degree) include Wezterm, Alacritty, kitty, foot, Ghostty, and iTerm2. Applications which support the kitty keyboard protocol (to some degree) include Vim, Neovim, Helix, kakoune, and nushell. This means that when using one of these applications in one of these terminals, all of the key encoding problems discussed above (as well as some others which were not discussed…) are solved.

Decorations

Xterm has supported 256 user specified colors since 1999. These colors could be changed at runtime using an escape sequence (OSC 4), which can be used to great effect (see “8 Bit & ‘8 Bitish’ Graphics-Outside the Box” by Mark Ferrari for an incredible demonstration, or install notcurses and run notcurses-demo j in your terminal).

Within the last decade or so, 24 bit color (sometimes referred to as “truecolor” or “RGB color”) has become widely supported by terminal emulators which allows terminal applications to use whatever arbitrary colors they want. This provides terminal UIs a much greater degree of flexibility and creative freedom.

Modern terminals also support other kinds of “rich” text markup, such as strikethrough and various types of underlines. For instance, text editors like Vim and Neovim can add a red squiggly line under misspelled words (as seen in many graphical rich text editors).

Examples of markup styles supported by modern terminal emulators

It is also possible to display images and even videos inline inside of terminal emulators. There are (at least) three different ways to do this (sixels, the iTerm2 image protocol, and the kitty graphics protocol) and support among terminal emulators varies. Unfortunately this means that terminal applications are in a bit of an awkward situation, as they must either implement support for all of the image protocols, or only support a subset of terminals. For this reason, use of images in terminal applications is still relatively uncommon.

It is important to note that advances in terminal based UIs are not only due to the efforts of terminal emulators, but also to the creativity and talent of terminal application and library authors. For example, see some of the fantastic work that charm.sh has done creating delightful, interactive terminal based user interfaces that rival (and in some cases, surpass!) graphical UIs for similar tools.

Capability Determination

Terminal emulators do not all support the same features. In some cases, the same feature is implemented in different ways. Terminal applications need some way to know which features the terminal they’re running in support and how to properly use those features.

Today this is primarily done using a distributed database of “terminfo” files. The terminal emulator uses the $TERM environment variable to communicate to terminal applications which terminfo file to use to lookup which capabilities the terminal supports.

This has a multitude of problems, however. The terminfo database is part of the ncurses library, and different operating systems and distributions package different versions of ncurses. This was a problem for tmux users on macOS for many years because the version of ncurses packaged with macOS was so old that it did not even include the tmux-256color terminfo entry at all!

This is also a problem for newer terminals which have not yet been added to the ncurses terminfo database. Terminal emulators can (and often do) ship their own terminfo entries which are used by applications running on the same system as the terminal emulator itself. But when connecting to a remote system (e.g. with SSH), the terminfo database on the remote system will not have the terminfo entry and the user is met with cryptic warnings like WARNING: terminal is not fully functional and applications not functioning properly.

To circumvent this issue, many terminals use xterm-256color as their $TERM value, essentially claiming to be Xterm even though they are not, piggybacking on Xterm’s ubiquity. This creates a vicious cycle, as terminal applications often hardcode special cases for xterm-256color, which incentivizes terminals to claim to be xterm-256color, which incentivizes applications to special case xterm-256color, which… and so on. The problem is exacerbated by common (bad) advice to users facing problems with terminal applications to simply override $TERM to be xterm-256color (the Xterm FAQ itself warns against this).

Unfortunately there are no easy fixes for these problems, but there is hope. The vast majority of escape sequences used by applications today are common across most (if not all) modern terminal emulators. This makes terminfo less necessary since applications can usually safely assume that a given escape sequence will “just work”.

In addition, terminal emulators increasingly support applications querying support for certain capabilities. For instance, applications can query the terminal for support of the kitty keyboard protocol mentioned above and only enable it if the terminal responds that it is supported. A nice property of escape sequence queries is they still work even over remote login connections like SSH.

Some new TUI libraries, such as vaxis, are designed specifically to avoid using terminfo at all and exclusively use queries to determine feature capabilities. As more applications, libraries, and terminal emulators move in this direction, terminfo will become increasingly unnecessary.

System Integration

One of the many advantages of software terminal emulators over hardware video terminals is that they are one piece of a larger, integrated computing system. Modern terminal emulators support many escape sequences to interact with their broader environment. These sequences are generally known as Operating System Commands (OSCs) and are often referred to by the numeric integer which appears after the OSC prefix.

Some of the more popular OSC sequences are OSC 2 for setting the title of the terminal emulator’s window (used frequently by shells and text editors), OSC 8 for creating clickable hyperlinks, OSC 9 for sending desktop notifications, and OSC 52 for interacting with the system clipboard.

You can test these sequences out for yourself. Try running the following in your shell:

printf '\e]9;This is a notification!\a'

If your terminal emulator supports OSC 9, you will see a desktop notification appear with the text, “This is a notification!” (some terminals or operating systems may not display a notificaton for the focused application. In that case, add a sleep 2 before the printf command and quickly change focus to another window).

Terminals which support OSC 8 can create clickable hyperlinks. For instance, try running the below command:

printf '\e]8;;https://www.youtube.com/watch?v=dQw4w9WgXcQ\aClick me for an awesome video!\n\e]8;;\a'

You will see the text “Click me for an awesome video!”. If your terminal emulator supports OSC 8, the text will be clickable (perhaps requiring a modifier key like Shift or Command to be held) and might be styled with an underline or some other visual affordance to indicate that the text is a hyperlink. Clicking on the text will open your web browser to the (perfectly innocuous) embedded URL.

A long standing issue for terminal based text editors like Vim is clipboard management in remote sessons. A strength of Vim is that it can be run just as easily in a remote SSH session as it can locally; however, the remote SSH session is not able to communicate with the clipboard on your local system, so it is not possible to copy text inside of Vim on the remote session to your clipboard.

Vim addresses this by (optionally) linking against X11 and allowing users to forward their X connection to the remote server, allowing Vim on the remote server to copy text to the X clipboard on the local system. And while this does work, it has its own problems (users must use a version of Vim compiled against X11, with the optional +clipboard feature enabled, and use X11 as their display server, and remember to forward the X connection to the remote system).

A better solution is to copy data to the clipboard through the terminal emulator directly. An application running in the terminal can use the OSC 52 escape sequence to write a Base64 encoded string to the terminal emulator. The terminal then decodes the string and copies the data into the system clipboard. The terminal emulator does not know or care whether the application that sent the sequence is running remotely or not, which means this works on any system with zero dependencies.

Pasting (reading) from the clipboard has serious security implications, because any program in the terminal (even ones on remote servers) can request the clipboard contents of the user’s system. For this reason, most terminal emulators disable reading from the clipboard by default, or require the user to explicitly allow it with a prompt.

Neovim recently added builtin support for using OSC 52 and it will be enabled for users by default (if the terminal emulator supports it) in the forthcoming 0.10 release.

Conclusion

While it’s true that terminals, as an application platform, are idiosyncratic and quirky, their portability, ubiquity, and relative ease of use (for application authors) makes them increasingly popular for many developers, even in the face of an increasing number of alternatives.

This article is not exhaustive, but it is not meant to be. There are other challenges that both terminal emulator and terminal application authors face that are not discussed here, as well as other areas of innovation and creative exploration. Some examples: better grapheme clustering, synchronized output to avoid “flickering” in redraw-heavy UIs, and custom shaders to create arbitrary visual effects.

Terminal emulators are not static: they continue to evolve and innovate to solve users’ problems and improve users’ experience. The underlying technology is old: downright ancient by the standards of modern tech. But, instead of a flaw, I consider this a strength: it gives me confidence that while individual terminal emulators may come and go, the underlying platform will endure.

References & Further Reading

This depends on your exact definition of “underlying technology”. Here, I’m referring to the use of teletype machines (TTYs) connected to a digital computer. But teletypes themselves were created well over a century ago, and there are still traces of their design in the inner workings of terminal emulators. ↩︎
Technically, it uses whichever version of the key, shifted or unshifted, falls in the range [0x20, 0x5f]. For alphabetic characters, this is the shifted (uppercase) letter, but for keys like [ or - it uses the unshifted variant. ↩︎

What's New in Neovim 0.7

Fri, 15 Apr 2022 07:14:00 -0600

Neovim 0.7 was just released, bringing with it lots of new features (and of course plenty of bug fixes). You can find the full release notes here, but in this post I’ll cover just a few of the new additions.

Lua everywhere!

Neovim 0.5 saw the introduction of Lua as a first-class citizen in the Neovim ecosystem: Lua could now be used in the user’s init file, plugins, colorschemes, ftplugins, etc. Basically, anywhere that you could use a .vim file, you could now use .lua instead.

However, there were still some shortcomings in the Lua API at that time. Notably absent was the ability to create autocommands in Lua, as well as bind key mappings directly to Lua functions. In order to do either of these things, users needed to resort to workarounds involving a round trip through Vimscript conversion, which is a bit clunky:

-- Using a Lua function in a key mapping prior to 0.7
local function say_hello()
    print("Hello world!")
end

_G.my_say_hello = say_hello

vim.api.nvim_set_keymap("n", "<leader>H", "<Cmd>call v:lua.my_say_hello()<CR>", {noremap = true})

The situation was similar for autocommands and custom user commands.

In Neovim 0.7, it is now possible to use all of the usual configuration primitives (key mappings, autocommands, user commands, etc.) directly in Lua, with no Vimscript conversion necessary. This also makes it possible to bind key mappings and autocommands directly to local Lua functions:

-- Using a Lua function in a key mapping in 0.7
vim.api.nvim_set_keymap("n", "<leader>H", "", {
    noremap = true,
    callback = function()
        print("Hello world!")
    end,
})

-- Creating an autocommand in 0.7
vim.api.nvim_create_autocmd("BufEnter", {
    pattern = "*",
    callback = function(args)
        print("Entered buffer " .. args.buf .. "!")
    end,
    desc = "Tell me when I enter a buffer",
})

-- Creating a custom user command in 0.7
vim.api.nvim_create_user_command("SayHello", function(args)
    print("Hello " .. args.args)
end, {
    nargs = "*",
    desc = "Say hi to someone",
})

You may notice that nvim_set_keymap must set the Lua callback as a key in the final table argument, while nvim_create_user_command can pass the callback function directly as a positional parameter. This is a consequence of Neovim’s strict API contract, which mandates that after an API function makes it into a stable release, it’s signature must not change in any way. However, because nvim_create_user_command is a new API function, we are able to add a bit of convenience by making its second argument accept either a string or a function.

Neovim 0.7 also includes a Lua-only convenience function vim.keymap.set for easily creating new key mappings:

vim.keymap.set("n", "<leader>H", function() print("Hello world!") end)

vim.keymap.set differs from nvim_set_keymap in the following ways:

It can accept either a string or a Lua function as its 3rd argument.
It sets noremap by default, as this is what users want 99% of the time.

The help docs contain much more information: run :h vim.keymap.set in Neovim to learn more.

Finally, users can now use the API function nvim_set_hl to modify global highlight groups (the equivalent of using :hi), opening the door to pure-Lua colorschemes.

Distinguishing modifier keys

Being a terminal based application, Neovim has long been subject to the constraints of terminal emulators, one of which being that many keys are encoded the same and thus indistinguishable to applications running in the terminal. For example, <Tab> and <C-I> use the same representation, as do <CR> and <C-M>. This has long meant that it is not possible to separately map <C-I> and <Tab>: mapping one necessarily maps both.

This has long been a point of annoyance and there are multiple solutions in the wild to address it. Neovim uses Paul Evans’ libtermkey, which in turn makes use of Evans’ own fixterms proposal for encoding modifier keys in an unambiguous way. As long as the terminal emulator controlling Neovim sends keys encoded in this way, Neovim can correctly interpret them.

Neovim 0.7 now correctly distinguishes these modifier key combos in its own input processing, so users can now map e.g. <Tab> and <C-I> separately. In addition, Neovim sends an escape sequence on startup that signals to the controlling terminal emulator that it supports this style of key encoding. Some terminal emulators (such as iTerm2, foot, and tmux) use this sequence to programatically enable the different encoding.

A note of warning: this cuts both ways! You may find that existing mappings to <Tab> or <C-I> (or <CR>/<C-M>) no longer work. The fix is easy, however; simply modify your mapping to use the actual key you want to use.

In addition to disambiguating these modifier pairs, this also enables new key mappings that were not possible before, such as <C-;> and <C-1>.

Support for this depends largely on the terminal you are using, so this will not affect all users.

Global statusline

Neovim 0.7 introduces a new “global” statusline, which can be enabled by setting laststatus=3. Instead of having one statusline per window, the global statusline always runs the full available width of Neovim’s containing window. This makes it useful to display information that does not change per-window, such as VCS information or the current working directory. Many statusline plugins are already making use of this new feature.

filetype.lua

In Neovim 0.7 there is a new (experimental) way to do filetype detection. A quick primer on filetype detection: when you first start Neovim it sources a file called filetype.vim in the $VIMRUNTIME directory. This file creates several hundred BufRead,BufNewFile autocommands whose sole purpose is to infer the filetype of the file based on information about the file, most commonly the file’s name or extension, but sometimes also using the file’s contents.

If you profile your startup time with nvim --startuptime you will notice that filetype.vim is one of the slowest files to load. This is because it is expensive to create so many autocommands. An alternative way to do filetype detection is to instead create one single autocommand that fires for every new buffer and then tries to match the filetype through a sequential series of steps. This is what the new filetype.lua does.

In addition to using a single autocommand, filetype.lua uses a table-based lookup structure, meaning that in many cases filetype detection happens in constant time. And if your Neovim is compiled with LuaJIT (which it most likely is), you also get the benefit of just-in-time compilation for this filetype matching.

This feature is currently opt-in as it does not yet completely match all of the filetypes covered by filetype.vim, although it is very close (I have been using it exclusively for many months without any issues). There are two ways to opt-in to this feature:

Use filetype.lua, but fallback to filetype.vim

Add let g:do_filetype_lua = 1 to your init.vim file. This prevents any regressions in filetype matching and ensures that filetypes are always detected at least as well as they are with filetype.vim. However, you will pay the startup time cost of both filetype.lua and filetype.vim.
Use only filetype.lua and do not load filetype.vim at all

Add both let g:do_filetype_lua = 1 and let g:did_load_filetypes = 0 to your init.vim. This will exclusively use filetype.lua for filetype matching and provides all of the performance benefits outlined above, with the (small) risk of missed filetype detection.

In addition to performance benefits, filetype.lua also makes it easy to add custom filetypes. Simply create a new file ~/.config/nvim/filetype.lua and call vim.filetype.add to create new matching rules. For example:

vim.filetype.add({
    extension = {
        foo = "fooscript",
    },
    filename = {
        ["Foofile"] = "fooscript",
    },
    pattern = {
        ["~/%.config/foo/.*"] = "fooscript",
    }
})

vim.filetype.add takes a table with 3 (optional) keys corresponding to “extension”, “filename”, and “pattern” matching. The value of each table entry can either be a string (in which case it is interpreted as the filetype) or a function. For example, you may want to override Neovim’s default behavior of always classifying .h files as C++ headers by using a heuristic that only sets the filetype to C++ if the header file includes another C++-style header (i.e. one without a trailing .h):

vim.filetype.add({
    extension = {
        h = function(path, bufnr)
            if vim.fn.search("\\C^#include <[^>.]\\+>$", "nw") ~= 0 then
                return "cpp"
            end
            return "c"
        end,
    },
})

We are bringing filetype.lua closer to full parity with filetype.vim every day. The goal is to make it the default in Neovim 0.8 (with the ability to opt-out to the traditional filetype.vim).

Client-server communication

Neovim 0.7 brings some of the features of neovim-remote into the core editor. You can now use nvim --remote to open a file in an already running instance of Neovim. An example:

# In one shell session
nvim --listen /tmp/nvim.sock

# In another shell session, opens foo.txt in the first Nvim instance
nvim --server /tmp/nvim.sock --remote foo.txt

One use case for the new remote functionality is the ability to open files from the embedded terminal emulator in the primary Neovim instance, rather than creating an embedded Neovim instance running inside Neovim itself.

Looking ahead to 0.8

Neovim is a loosely structured project of motivated individuals who do the work for fun; thus, any roadmap is always a bit of a guessing game. However, there are some things already brewing that you might see in Neovim 0.8:

Improvements to Treesitter support
“Projects” support for LSP
Anti-conceal
Fully remote TUI
And more…

Exploring Mach-O, Part 4

Sun, 16 Jan 2022 19:13:53 -0700

This is part 4 of a 4 part series exploring the structure of the Mach-O file format. Here are links to part 1, part 2, and part 3.

When we started this series we didn’t know anything about Mach-O other than some vague idea that it’s a binary file format used by macOS. By now, we have a much better understanding of how Mach-O is laid out and how the operating system uses the information in a Mach-O file to run a program.

We learned that every Mach-O file has a 32 byte header right at the beginning. The first 4 bytes of the header (and thus, the first 4 bytes of every Mach-O) file are a magic number that indicates that the file is, in fact, encoded using Mach-O. Much to our delight, we found that this magic number is 0xfeedface on 32-bit systems, and 0xfeedfacf on 64-bit systems.

This header tells us a lot more about our file too, including the type of CPU it is compiled for and the size of the load commands that come directly after it.

After the header come a sequence of load commands. Every load command shares two fields in common: the command type (which we encoded as a Command enum in Zig) and the total size in bytes of the command. Each command is parsed slightly differently, and some commands (such as the segment_commands) have more data structures that come directly after them.

There are a lot of different load commands that a Mach-O file can contain: our Command enum has 54 different values. Not every Mach-O file contains all of these commands, of course. We primarily investigated the Segment commands, which are an important concept for understanding how a Mach-O file is laid out. Each segment can have zero or more sections, and in our tiny exit program we found 3 segments: __PAGEZERO, __TEXT, and __LINKEDIT. More complex programs will have more segments, notably a __DATA segment, which our program lacks.

Immediately following the load commands are the segments. We found that our file contains a large block of contiguous zeros, and we discovered that this is because the header and load commands share the first segment with the __TEXT segment. The sections within the __TEXT segment are aligned to the end of the segment, and because segments must be page aligned, there can often be large chunks of unused space between the end of the load commands and the start of the first section in the __TEXT segment. This problem is “worse” on systems with larger page sizes, such as ARM64 (which uses a 16 KiB page size rather than the standard 4 KiB pages on x86).

We learned how to use Zig to parse binary file formats, such as using packed structs to represent bit fields and validating enum values using the std.meta.intToEnum standard library function. We also explored some cool parts of Zig’s comptime features to make an elegant and easy-to-use API for our parser.

But wait, there’s more

If you followed along then you know there is a lot we didn’t cover. There are still a lot of load commands we didn’t even talk about, and who knows what kind of goodies those involve.

We also didn’t talk about universal binaries. Apple has transitioned between ISAs twice: first from PowerPC to Intel x86, and then again to ARM. Because of this, they figured out a long time ago how to combine two or more binary formats into a single file. Apple refers to this as a “universal binary”, which is a packaged archive of multiple Mach-O files. Universal binaries have their own format, including their own header and magic number (spoiler alert: the universal binary magic number is 0xcafebabe. Is that even better than 0xfeedface?).

The point is, there are many rabbit holes for you to follow if you’re interested. I hope you do, and if you do, I hope you write about!

Thanks for reading!

Exploring Mach-O, Part 3

Sun, 16 Jan 2022 19:13:51 -0700

This is part 3 of a 4 part series exploring the structure of the Mach-O file format. Here are links to part 1, part 2, and part 4.

We left off with a basic parser that is able to parse the Mach-O header from our file. Now that the bones of our parser are fleshed out, it should be fairly straightforward to parse the rest of the file.

Load commands

Let’s take a look at our header again:

$ zig-out/bin/macho ../exit
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 16
sizeofcmds: 744
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE

Our header tells us that this Mach-O file has 16 load commands. Each load command shares two common fields:

struct load_comand {
	uint32_t cmd;
	uint32_t cmdsize;
};

The list of possible load commands is defined in mach-o/loader.h. Here are just a few:

/* Constants for the cmd field of all load commands, the type */
#define	LC_SEGMENT	0x1	/* segment of this file to be mapped */
#define	LC_SYMTAB	0x2	/* link-edit stab symbol table info */
#define	LC_SYMSEG	0x3	/* link-edit gdb symbol table info (obsolete) */
#define	LC_THREAD	0x4	/* thread */
#define	LC_UNIXTHREAD	0x5	/* unix thread (includes a stack) */
#define	LC_LOADFVMLIB	0x6	/* load a specified fixed VM shared library */
#define	LC_IDFVMLIB	0x7	/* fixed VM shared library identification */
#define	LC_IDENT	0x8	/* object identification info (obsolete) */

Each of these load commands, in turn, has its own struct representation. For example, here is the definition for the LC_SEGMENT_64 load command:

/*
 * The 64-bit segment load command indicates that a part of this file is to be
 * mapped into a 64-bit task's address space.  If the 64-bit segment has
 * sections then section_64 structures directly follow the 64-bit segment
 * command and their size is reflected in cmdsize.
 */
struct segment_command_64 { /* for 64-bit architectures */
	uint32_t	cmd;		/* LC_SEGMENT_64 */
	uint32_t	cmdsize;	/* includes sizeof section_64 structs */
	char		segname[16];	/* segment name */
	uint64_t	vmaddr;		/* memory address of this segment */
	uint64_t	vmsize;		/* memory size of this segment */
	uint64_t	fileoff;	/* file offset of this segment */
	uint64_t	filesize;	/* amount to map from the file */
	vm_prot_t	maxprot;	/* maximum VM protection */
	vm_prot_t	initprot;	/* initial VM protection */
	uint32_t	nsects;		/* number of sections in segment */
	uint32_t	flags;		/* flags */
};

Converting every load command into a Zig struct is pretty tedious work. If we were creating a production-ready Mach-O parser, it’s something we would need to do. However, since we’re just exploring, we can be lazy and just implement the load commands that are actually present in our Mach-O file.

First, let’s define a struct for the shared load command fields:

/// src/macho.zig

pub const LoadCommand = struct {
    /// Field definitions
    cmd: Command,
    cmdsize: u32,
};

Next we need to define the list of possible commands in a Command enum. Again, feel free to simply copy and paste if you’re following along:

/// src/macho.zig

const Command = enum(u32) {
    segment = 0x1,
    symtab = 0x2,
    symseg = 0x3,
    thread = 0x4,
    unixthread = 0x5,
    loadfvmlib = 0x6,
    idfvmlib = 0x7,
    ident = 0x8,
    fvmfile = 0x9,
    prepage = 0xa,
    dysymtab = 0xb,
    load_dylib = 0xc,
    id_dylib = 0xd,
    load_dylinker = 0xe,
    id_dylinker = 0xf,
    prebound_dylib = 0x10,
    routines = 0x11,
    sub_framework = 0x12,
    sub_umbrella = 0x13,
    sub_client = 0x14,
    sub_library = 0x15,
    twolevel_hints = 0x16,
    prebind_cksum = 0x17,
    load_weak_dylib = (0x18 | loader.LC_REQ_DYLD),
    segment_64 = 0x19,
    routines_64 = 0x1a,
    uuid = 0x1b,
    rpath = (0x1c | loader.LC_REQ_DYLD),
    code_signature = 0x1d,
    segment_split_info = 0x1e,
    reexport_dylib = (0x1f | loader.LC_REQ_DYLD),
    lazy_load_dylib = 0x20,
    encryption_info = 0x21,
    dyld_info = 0x22,
    dyld_info_only = (0x22 | loader.LC_REQ_DYLD),
    load_upward_dylib = (0x23 | loader.LC_REQ_DYLD),
    version_min_macosx = 0x24,
    version_min_iphoneos = 0x25,
    function_starts = 0x26,
    dyld_environment = 0x27,
    main = (0x28 | loader.LC_REQ_DYLD),
    data_in_code = 0x29,
    source_version = 0x2A,
    dylib_code_sign_drs = 0x2B,
    encryption_info_64 = 0x2C,
    linker_option = 0x2D,
    linker_optimization_hint = 0x2E,
    version_min_tvos = 0x2F,
    version_min_watchos = 0x30,
    note = 0x31,
    build_version = 0x32,
    dyld_exports_trie = (0x33 | loader.LC_REQ_DYLD),
    dyld_chained_fixups = (0x34 | loader.LC_REQ_DYLD),
    fileset_entry = (0x35 | loader.LC_REQ_DYLD),
};

Just like our MachHeader64 struct, let’s add a parse function to LoadCommand that tells our parser how this structure should be parsed.

/// src/macho.zig

pub const LoadCommand = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(parser: *Parser) !LoadCommand {
        const cmd = try parser.parse(Command);
        const cmdsize = try parser.parse(u32);

        std.debug.print("{}, size: {d}\n", .{cmd, cmdsize});

        return LoadCommand{
            .cmd = cmd,
            .cmdsize = cmdsize,
        };
    }
};

We can see that our parser API makes this pretty simple. We no longer need to think about how a Command enum is parsed: we just tell our parser to do it. Similarly, we can now simply use

parser.parse(LoadCommand)

to parse a LoadCommand struct. No fuss.

Let’s go ahead and add that to main.zig.

/// src/main.zig

var parser = macho.Parser.init(data);

const header = try parser.parse(macho.MachHeader64);
std.debug.print("{}\n", .{header});

var i: usize = 0;
while (i < header.ncmds) : (i += 1) {
    _ = try parser.parse(macho.LoadCommand);
}

We know the number of load commands from the header, so we just need to read each one in a loop. Since we are not using the return value from parse (yet), we need to mark the variable as unused with _.

If we try to build and run this, we hit our first error:

$ zig build run -- ../exit
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 16
sizeofcmds: 744
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE
Command.segment_64, size: 72
1095786335 is not a valid value for Command
error: InvalidEnumTag
/opt/zig/lib/zig/std/meta.zig:823:5: 0x102972097 in std.meta.intToEnum (macho)
    return error.InvalidEnumTag;
    ^
/Users/greg/src/gpanders.com/macho/src/macho.zig:44:21: 0x102971633 in macho.Parser
.parse (macho)
                    return err;
                    ^
/Users/greg/src/gpanders.com/macho/src/macho.zig:224:21: 0x102971493 in macho.LoadC
ommand.parse (macho)
        const cmd = try parser.parse(Command);
                    ^
/Users/greg/src/gpanders.com/macho/src/macho.zig:26:20: 0x10296e25f in macho.Parser
.parse (macho)
            return try T.parse(self);
                   ^
/Users/greg/src/gpanders.com/macho/src/main.zig:40:30: 0x10296d7c7 in main (macho)
        const load_command = try parser.parse(macho.LoadCommand);
                             ^

It looks like we tried to convert an invalid int (1095786335) into our Command enum. That enum doesn’t have a value that corresponds to that number, so std.meta.intToEnum threw an error.

Do you see where we went wrong? We tried parsing load commands successively, as if they were neatly laid out in an contiguous array in the Mach-O file. However, the file format reference tells us that each LoadCommand struct is followed by more fields depending on the type of command. Further, individual commands may themselves be followed by more data.

The LoadCommand struct conveniently tells us the size in bytes of each load command. For now, we can simply use this number to skip over the rest of the load command so that we can read each of the load command headers present in the file.

Let’s add a skip method to our parser:

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn init(...) { ... }
    pub fn parse(...) { ... }

    pub fn skip(self: *Parser, n: usize) !void {
        if (self.data.len < n) {
            return error.NotEnoughBytes;
        }

        self.data = self.data[n..];
    }
};

This is a pretty simple function. Hopefully it’s self explanatory.

Now let’s add this to LoadCommand.parse:

/// src/macho.zig

pub const LoadCommand = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(parser: *Parser) !LoadCommand {
        const cmd = try parser.parse(Command);
        const cmdsize = try parser.parse(u32);

        try parser.skip(cmdsize);

        return LoadCommand{
            .cmd = cmd,
            .cmdsize = cmdsize,
        };
    }
};

Let’s build and run:

$ zig build run -- ../exit
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 16
sizeofcmds: 744
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE
Command.segment_64, size: 72
1163157343 is not a valid value for Command
error: InvalidEnumTag
/opt/zig/lib/zig/std/meta.zig:823:5: 0x10492209b in std.meta.intToEnum (macho)
    return error.InvalidEnumTag;
    ^
/Users/greg/src/gpanders.com/macho/src/macho.zig:44:21: 0x104921637 in macho.Parser
.parse (macho)
                    return err;
                    ^
/Users/greg/src/gpanders.com/macho/src/macho.zig:224:21: 0x104921497 in macho.LoadC
ommand.parse (macho)
        const cmd = try parser.parse(Command);
                    ^
/Users/greg/src/gpanders.com/macho/src/macho.zig:26:20: 0x10491e18b in macho.Parser
.parse (macho)
            return try T.parse(self);
                   ^
/Users/greg/src/gpanders.com/macho/src/main.zig:40:30: 0x10491d67f in main (macho)
        const load_command = try parser.parse(macho.LoadCommand);
                             ^

Agh! We are still getting an “invalid value for Command” error. What gives?

There is a subtle mistake in our logic. Do you see it? The cmdsize field of the LoadCommand struct gives the total size of the load command in bytes, including the common fields in the LoadCommand struct itself. When we skip over cmdsize bytes, we are skipping too far, because we are counting the 8 bytes of the LoadCommand struct twice.

Easy fix:

try parser.skip(cmdsize - @sizeOf(LoadCommand));

Let’s try again:

$ zig build run -- ../exit
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 16
sizeofcmds: 744
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE
Command.segment_64, size: 72
Command.segment_64, size: 232
Command.segment_64, size: 72
Command.dyld_chained_fixups, size: 16
Command.dyld_exports_trie, size: 16
Command.symtab, size: 24
Command.dysymtab, size: 80
Command.load_dylinker, size: 32
Command.uuid, size: 24
Command.build_version, size: 32
Command.source_version, size: 16
Command.main, size: 24
Command.load_dylib, size: 56
Command.function_starts, size: 16
Command.data_in_code, size: 16
Command.code_signature, size: 16

Look at that! We now have a list of all of the load commands present in our Mach-O file.

We’re not necessarily interested in parsing all of these load commands, but we’ll certainly want to take a deeper look at the segment_64 sections. The commands we don’t care about we can simply skip over, but for those we do, we will need to define structs to represent their data.

Let’s start with segment_64. The Mach-O headers define the following struct:

struct segment_command_64 { /* for 64-bit architectures */
	uint32_t	cmd;		/* LC_SEGMENT_64 */
	uint32_t	cmdsize;	/* includes sizeof section_64 structs */
	char		segname[16];	/* segment name */
	uint64_t	vmaddr;		/* memory address of this segment */
	uint64_t	vmsize;		/* memory size of this segment */
	uint64_t	fileoff;	/* file offset of this segment */
	uint64_t	filesize;	/* amount to map from the file */
	vm_prot_t	maxprot;	/* maximum VM protection */
	vm_prot_t	initprot;	/* initial VM protection */
	uint32_t	nsects;		/* number of sections in segment */
	uint32_t	flags;		/* flags */
};

We will omit the first two common fields from our struct to arrive at:

/// src/macho.zig

const SegmentCommand64 = packed struct {
    segname: [16]u8,
    vmaddr: u64,
    vmsize: u64,
    fileoff: u64,
    filesize: u64,
    maxprot: VmProt,
    initprot: VmProt,
    nsects: u32,
    flags: SegmentCommandFlags,
};

Notice that we mark this struct as packed. This forces the memory layout of this struct to match the order that we specified here. This will allow us to safely parse an array of bytes into this struct without needing to define a parse() function and have each field line up nicely. We don’t need to use a parse() function in this case since there are no enum fields that we need to validate.

This also requires us to define the VmProt and SegmentCommandFlags structs. These are both bitfields, so we again use packed structs with boolean fields:

/// src/macho.zig

const VmProt = packed struct {
    read: bool,
    write: bool,
    execute: bool,
    _: u29, // pad to 32 bits
};

const SegmentCommandFlags = packed struct {
    highvm: bool,
    fvmlib: bool,
    noreloc: bool,
    protected_version_1: bool,
    read_only: bool,
    _: u27, // pad to 32 bits
};

Each segment contains a number of sections, determined by the nsects field of SegmentCommand64. A section looks like this:

/// src/macho.zig

const Section64 = packed struct {
    sectname: [16]u8,
    segname: [16]u8,
    addr: u64,
    size: u64,
    offset: u32,
    @"align": u32,
    reloff: u32,
    nreloc: u32,
    flags: u32,
    reserved1: u32,
    reserved2: u32,
    reserved3: u32,
};

Note the syntax for @"align". This is necessary because align is a keyword in Zig. The @"" syntax lets us use arbitrary identifiers for variables and struct fields, which gets us around this restriction.

Similar to what we did with MachHeader64 we can implement a public format function for SegmentCommand64 and Section64 to improve how these structs are printed out (I leave this as an exercise for the reader).

Let’s update LoadCommand.parse with these new data structures:

/// src/macho.zig

pub const LoadCommand = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(parser: *Parser) !LoadCommand {
        const cmd = try parser.parse(Command);
        const cmdsize = try parser.parse(u32);

        switch (cmd) {
            .segment_64 => {
                const segment_command_64 = try parser.parse(SegmentCommand64);
                std.debug.print("{}\n", .{segment_command_64});

                var i: usize = 0;
                while (i < segment_command_64.nsects) : (i += 1) {
                    const section = try parser.parse(Section64);
                    std.debug.print("{}\n", .{section});
                }
            },
            else => try parser.skip(cmdsize - @sizeOf(LoadCommand)),
        }

        return LoadCommand{
            .cmd = cmd,
            .cmdsize = cmdsize,
        };
    }
};

When we build and run this, we see a lot of information about our executable’s segments and sections!

$ zig build run -- ../exit
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 16
sizeofcmds: 744
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE
Command.segment_64, size: 72
segname: __PAGEZERO
vmaddr: 0x0
vmsize: 0x100000000
fileoff: 0x0
filesize: 0x0
maxprot: VmProt{ .read = false, .write = false, .execute = false, ._ = 0 }
initprot: VmProt{ .read = false, .write = false, .execute = false, ._ = 0 }
nsects: 0
flags:
Command.segment_64, size: 232
segname: __TEXT
vmaddr: 0x100000000
vmsize: 0x4000
fileoff: 0x0
filesize: 0x4000
maxprot: VmProt{ .read = true, .write = false, .execute = true, ._ = 0 }
initprot: VmProt{ .read = true, .write = false, .execute = true, ._ = 0 }
nsects: 2
flags:
segname: __TEXT
sectname: __text
addr: 0x100003fa0
size: 0xc
offset: 16288
align: 2^4 (16)
reloff: 0
nreloc: 0
flags: 0x80000400

segname: __TEXT
sectname: __unwind_info
addr: 0x100003fac
size: 0x48
offset: 16300
align: 2^2 (4)
reloff: 0
nreloc: 0
flags: 0x0

Command.segment_64, size: 72
segname: __LINKEDIT
vmaddr: 0x100004000
vmsize: 0x4000
fileoff: 0x4000
filesize: 0x1c1
maxprot: VmProt{ .read = true, .write = false, .execute = false, ._ = 0 }
initprot: VmProt{ .read = true, .write = false, .execute = false, ._ = 0 }
nsects: 0
flags:
Command.dyld_chained_fixups, size: 16
Command.dyld_exports_trie, size: 16
Command.symtab, size: 24
Command.dysymtab, size: 80
Command.load_dylinker, size: 32
Command.uuid, size: 24
Command.build_version, size: 32
Command.source_version, size: 16
Command.main, size: 24
Command.load_dylib, size: 56
Command.function_starts, size: 16
Command.data_in_code, size: 16
Command.code_signature, size: 16

The first segment is called __PAGEZERO and has no sections. It also has both its fileoff and filesize fields set to 0, suggesting that the __PAGEZERO segment is not actually present in the on-disk file. Rather, it represents a region of virtual memory that the operating system will create when the program is loaded. Indeed, we see that this segment’s vmaddr field is 0 and its vmsize field is 0x100000000. This means that the first 0x100000000 bytes of virtual memory in our program will simply be zero. This is done intentionally to catch null pointer dereferences.

Next is the __TEXT segment. This segment’s fileoff field is also 0, but this time it has a non-zero size of 0x4000 or 16 KiB. This tells us that the __TEXT segment starts at the very beginning of the file. We’ll come back to this a little bit later.

The vmaddr of the __TEXT segment starts at 0x100000000, immediately after the __PAGEZERO segment. When the operating system loads this program, it will map the region between 0x0 and 0x4000 of this file to the virtual memory range between 0x100000000 and 0x100004000. We also see that the protection flags for this segment are set to read and execute, but not write, which is what we would expect from a section containing executable machine instructions.

The __TEXT segment contains two sections: __text and __unwind_info. Hey, __TEXT,__text sure looks familiar…

$ objdump -d ../exit

../exit:        file format mach-o arm64


Disassembly of section __TEXT,__text:

0000000100003fa0 <_main>:
100003fa0: 40 05 80 d2  mov     x0, #42
100003fa4: 30 00 80 d2  mov     x16, #1
100003fa8: 01 10 00 d4  svc     #0x80

That’s right, that’s the text section of our binary and where the actual machine instructions live! According to our macho parser, the __text section is 0xc (12) bytes long and starts at offset 16288 in our file. Let’s check this for ourselves with xxd:

$ xxd -s 16288 -l 12 ../exit
00003fa0: 4005 80d2 3000 80d2 0110 00d4            @...0.......

Those are the machine instructions, exactly as we expect.

Going further

Back in the beginning I mentioned that a Mach-O file has the basic structure of a header, followed by load commands, followed by segments. If we continue to parse bytes past the last load command, what will we find?

First, it will be useful to keep track of how many bytes we’ve parsed in our parser, which will tell us the current offset in the file. Let’s add a new offset field to the Parser struct and update it in the parse() and skip() functions:

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    data: []const u8,
    offset: usize = 0, // NEW!

    /// Functions
    pub fn parse(self: *Parser, comptime T: type) !T {
        ...

        self.data = self.data[size..];
        self.offset += size; // NEW!
        return val;
    }

    pub fn skip(self: *Parser, n: usize) !void {
        ...

        self.data = self.data[n..];
        self.offset += n; // NEW!
    }
};

And now everywhere we print a parsed value we can also print the parser’s offset:

/// src/macho.zig

pub const LoadCommand = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(parser: *Parser) !LoadCommand {
        const cmd = try parser.parse(Command);
        const cmdsize = try parser.parse(u32);

        std.debug.print("0x{x}: {}, size: {d}\n", .{
            parser.offset,
            cmd,
            cmdsize,
        });

        switch (cmd) {
            .segment_64 => {
                const segment_command_64 = try parser.parse(SegmentCommand64);
                std.debug.print("0x{x}: {}\n", .{ parser.offset, segment_command_64 });

                var j: usize = 0;
                while (j < segment_command_64.nsects) : (j += 1) {
                    const section = try parser.parse(Section64);
                    std.debug.print("0x{x}: {}\n", .{ parser.offset, section });
                }
            },
            else => try parser.skip(cmdsize - @sizeOf(LoadCommand)),
        }

        return LoadCommand{
            .cmd = cmd,
            .cmdsize = cmdsize,
        };
    }
};

When we run this we can see how much space the header and load commands take together:

$ zig build run -- ../exit
# truncated output...
0x1a0: Command.dyld_chained_fixups, size: 16
0x1b0: Command.dyld_exports_trie, size: 16
0x1c0: Command.symtab, size: 24
0x1d8: Command.dysymtab, size: 80
0x228: Command.load_dylinker, size: 32
0x248: Command.uuid, size: 24
0x260: Command.build_version, size: 32
0x280: Command.source_version, size: 16
0x290: Command.main, size: 24
0x2a0: EntryPointCommand{ .entryoff = 16288, .stacksize = 0 }
0x2a8: Command.load_dylib, size: 56
0x2e0: Command.function_starts, size: 16
0x2f0: Command.data_in_code, size: 16
0x300: Command.code_signature, size: 16

The last load command starts at 0x300 and then takes another 8 bytes (16 bytes is the command size, minus 8 bytes for the common fields). This puts us at offset 776 (0x308). But of course, we already knew this because the header told us that the load commands were 744 bytes long, and the header itself is 32 bytes.

So what comes after 0x308?

$ xxd -s $((0x308)) -l 32 ../exit
00000308: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000318: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Hmm, just a bunch of zeros. Let’s try reading some more:

$ xxd -s $((0x308)) -l 64 ../exit
00000308: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000318: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000328: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000338: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Still just zeros! How many zeros are there exactly? Let’s count:

/// src/main.zig

var i: usize = 0;
while (i < header.ncmds) : (i += 1) {
    _ = try parser.parse(macho.LoadCommand);
}

// NEW!
while (true) {
    var word = try parser.parse(u32);
    if (word != 0) break;
}

std.debug.print("Next non-zero byte starts at 0x{x}\n", .{parser.offset - @sizeOf(u32)});

$ zig build run -- ../exit
Next non-zero byte starts at 0x3fa0

So every byte from 0x308 to 0x3fa0 is zero (we recognize 0x3fa0 as the start of the __text section). That’s 15512 bytes worth of zeros! What’s going on?

The Mach-O segments are page aligned and the page size on ARM64 macOS is 16 KiB (0x4000). The header and load commands are considered part of the first segment, which in this case is the __TEXT segment¹. This makes sense, since the fileoff field (which represents the start offset in the actual on-disk file of the segment) the __TEXT segment is 0, which is exactly where the header is.

I could not find any official documentation stating this, but my hypothesis is that the starting offset of the first section in the __TEXT segment is calculated by summing the size of all of the sections within the __TEXT segment and then subtracting that from the nearest page boundary. The space between the end of the load commands and the first section (__text) is then filled with zeros.

For example, our exit program has two sections in the __TEXT segment: __text and __unwind_info. The sizes of these sections are 12 (0xc) bytes and 72 (0x48) bytes respectively, or 84 bytes total. The total size of all of the load commands is 744 bytes (from the header), plus the 32 bytes of the header itself. Adding all of these up we get 84 + 744 + 32 = 860. Rounding to the nearest page boundary gets us to 16384 (0x4000). Subtracting the size of the __unwind_info section (72) puts us at 0x3fb8, which is 12 bytes off from the actual start of the __unwind_info section, 0x3fac. To be honest I’m not sure where that 12 byte difference is coming from. If you know, please send me an email.

We can check this theory against another Mach-O file: the macho executable itself. If we run macho on macho we see the following information (filtered out to only the relevant parts):

$ zig-out/bin/macho zig-out/bin/macho
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 17
sizeofcmds: 1784
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE HAS_TLV_DESCRIPTORS
0x28: Command.segment_64, size: 72
0xb0: segname: __TEXT
vmaddr: 0x100000000
vmsize: 0x90000
fileoff: 0x0
filesize: 0x90000
maxprot: VmProt{ .read = true, .write = false, .execute = true, ._ = 0 }
initprot: VmProt{ .read = true, .write = false, .execute = true, ._ = 0 }
nsects: 5
flags:
0x100: segname: __TEXT
sectname: __text
addr: 0x1000018f8
size: 0x7ed54
offset: 6392
align: 2^2 (4)
reloff: 0
nreloc: 0
flags: 0x80000400

0x150: segname: __TEXT
sectname: __stubs
addr: 0x10008064c
size: 0x138
offset: 525900
align: 2^2 (4)
reloff: 0
nreloc: 0
flags: 0x80000408

0x1a0: segname: __TEXT
sectname: __stub_helper
addr: 0x100080784
size: 0x150
offset: 526212
align: 2^2 (4)
reloff: 0
nreloc: 0
flags: 0x80000400

0x1f0: segname: __TEXT
sectname: __cstring
addr: 0x1000808d4
size: 0x987
offset: 526548
align: 2^0 (1)
reloff: 0
nreloc: 0
flags: 0x2

0x240: segname: __TEXT
sectname: __const
addr: 0x100081260
size: 0xeda0
offset: 528992
align: 2^4 (16)
reloff: 0
nreloc: 0
flags: 0x0

Next non-zero byte starts at 0x18f8

The size of the commands is 1784 bytes, the size of the __text section is 519508 (0x7ed54) bytes, __stubs is 312 (0x138) bytes, __stub_helper is 336 (0x150) bytes, __cstring is 2439 (0x987) bytes, and __const is 60832 (0xeda0) bytes. Combined, the __TEXT sections are 583427 (0x8e703) bytes. When we add the 1784 bytes from the load commands and 32 bytes from the header, the total size of the first segment is 585243 (0x8ee1b) bytes. The nearest page boundary is then 0x90000, which is what we see for the filesize field of the __TEXT segment.

If we subtract 0x8e703 from 0x90000 we get 0x18fd. This is close to the first non-zero byte at 0x18f8, but doesn’t quite match. In this case, however, we know why: each section within the __TEXT segment has its own alignment, so there are some wasted bytes in between the different sections to maintain that alignment. When we account for these bytes, we get the expected start value of 0x18f8.

So that explains the mystery zeros: page alignment!

Let’s go back to our exit program. After the __TEXT segment is __LINKEDIT, which is mandated to always be the last segment. Because segments are always aligned to page boundaries, this segment begins at 0x4000. According to the load command, this segment is 449 (0x1c1) bytes long. The Mach-O file format reference tells us that the __LINKEDIT segment contains “raw data used by the dynamic linker, such as symbol, string, and relocation table entries”.

This is the very end of our file, which we can confirm with stat:

$ stat -f '%#Xz' exit
0x41c1

We’ve reached the end of our Mach-O file! The file was arranged exactly as we expected: a header, followed by a sequence of load commands, followed by the actual segments. We ignored most of the load commands, and our simple program only had two actual segments, so there wasn’t much to look at. In a larger or more complex program, we would find a lot more.

In the next and final part, we’ll do a recap of what we learned and discuss some other things we could try if we wanted to dig further.

Technically, __PAGEZERO is the first segment, which maps an empty region of zeros in the virtual address range 0x0 to 0x100000000. However, this segment takes no space in the on-disk file, so the “first segment” is actually the segment that comes after it (__TEXT). ↩︎

Exploring Mach-O, Part 2

Sun, 16 Jan 2022 19:13:50 -0700

This is part 2 of a 4 part series exploring the structure of the Mach-O file format. Here are links to part 1, part 3, and part 4.

Last time, we created our own tiny Mach-O executable. This program doesn’t do anything useful, it’s simply the smallest executable we can use to examine what the Mach-O file format looks like.

From here on, we’ll write our own primitive parser to examine the contents of our program. I’m going to write my Mach-O parser in Zig. Why Zig? Mostly because I really like it and I find it’s quite easy to get simple things like this up and running. It’s also particularly well suited to tasks like this¹.

Note to readers in the future: it’s important to note that Zig does not yet have a stable 1.0 release. While at this point the language itself is fairly stable, the standard library often has breaking changes. I’ll do my best to keep the code in this article up-to-date, but be warned that it may not work by the time you read it. You can find the full source code on sourcehut.

First things first, let’s bootstrap an executable:

$ mkdir macho
$ cd macho
$ zig init-exe
info: Created build.zig
info: Created src/main.zig
info: Next, try `zig build --help` or `zig build run`

We’ll leave main.zig simple and simply call our parsing function and then print the parsed data. We’ll put the guts of our parser in src/macho.zig.

/// src/macho.zig

// First, import std, cuz we're gonna need it
const std = @import("std");

// Now, let's create a Parser struct
pub const Parser = struct {
    /// Field definitions
    // Our parser will hold a slice of bytes
    data: []const u8,

    /// Functions
    ...
};

To start off, we define our Parser struct with a single field: a slice of bytes (or u8 in Zig). We mark this slice as const because we don’t plan to mutate the data, we are simply interpreting it.

Our parser will work by implementing a few parse functions to read off a certain number of bytes from the front of the data slice and interpret those bytes as a given value. Let’s first add an init function to initialize a Parser object from an array of bytes.

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn init(data: []const u8) Parser {
        return Parser{
            .data = data,
        };
    }
};

Next, let’s add a simple parseLiteral function:

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn init(...) { ... }

    pub fn parseLiteral(self: *Parser, comptime T: type) !T {

    }
};

Let’s explain what’s going on here. The first argument to our function is a pointer to a *Parser object. Because this function is defined within the Parser struct itself, Zig treats this argument as a “receiver”, meaning we can use standard method call syntax (e.g. parser.parseLiteral()). The object on which this method is called is used as the first argument (self). We use a pointer to Parser because this function will mutate the parser object (by removing bytes from the data byte slice).

Next, the second argument is a comptime argument, which means it has to be known at compile time. It also has type type, which may be confusing at first. In Zig’s comptime, types are just values, which means you can do things like

const MySuperCoolType = u32;
const y: MySuperCoolType = 42;

This also means that we can accept a type as an argument to our function. This is how Zig does polymorphism. In our case, we accept a type T which is also the return type. So this function will read some bytes off the front of our data byte slice, interpret those bytes as a T, and then return the value.

Finally, the return type !T means that this function returns a T or an error. If you’re familiar with Rust, this is similar to Result<T, Error>.

This is what the implementation looks like:

    pub fn parseLiteral(self: *Parser, comptime T: type) !T {
        const size = @sizeOf(T);
        if (self.data.len < size) {
            return error.NotEnoughBytes;
        }

        const bytes = self.data[0..size];
        self.data = self.data[size..];
        return std.mem.bytesToValue(T, bytes);
    }

First, we use the builtin @sizeOf function to get the size of the type T in bytes. We then ensure that our byte slice has enough data in it: if it does not, we return a NotEnoughBytes error.

We then grab size bytes from our byte slice and then mutate the byte slice to remove those bytes from the front. We then call std.mem.bytesToValue(T, bytes) to re-interpret those bytes as a type T.

Is this safe to do? When we’re parsing integers (which we’ll be doing a lot of), this is fine, so long as the bytes are in the endian order we expect. On macOS, everything is little endian, so this is not an issue. We can also parse structs that are made up strictly of integers or arrays of integers for the same reason.

There are also no lifetime concerns here: bytesToValue creates a copy of the bytes being interpreted, but even if it didn’t, the byte slice that our parser is operating under has the same lifetime as our program since it is created in main() and is not released until the program exits.

If we want to parse an enum, then we need to validate that the value we’re parsing is a valid enum value. We will do this later when we introdue a parseEnum function.

In our main.zig file, we can test this out by adding some boilerplate to open and mmap a file:

/// src/main.zig

const std = @import("std");

const macho = @import("macho.zig");

pub fn main() anyerror!void {
    // Read the first command line argument. If it doesn't exist, return an
    // error
    var args = std.process.args();
    _ = args.skip();
    const fname = args.nextPosix() orelse {
        std.debug.print("Missing required argument: FILENAME\n", .{});
        return error.MissingArgument;
    };

    // Open the file. We use `defer` to ensure the file is closed when the
    // variable goes out of scope. The `try` keyword is semantic sugar that
    // uses the result of the function if no error occurs; otherwise, it
    // returns whatever error value the function itself returned (if you're
    // familiar with Rust, this is like the `?` operator).
    var file = try std.fs.cwd().openFile(fname, .{});
    defer file.close();

    // This is a standard mmap(2) call. If you're unfamiliar with mmap, give
    // `man 2 mmap` a read. This memory maps the file's contents into our
    // program's virtual memory space. This gives us access to the bytes
    // without having to copy them. Again, we use `defer` to "clean up" the
    // mmap when `data` goes out of scope.
    const data = try std.os.mmap(null, try file.getEndPos(), std.os.PROT.READ, std.os.MAP.PRIVATE, file.handle, 0);
    defer std.os.munmap(data);

    // Finally, we initialize our parser.
    var parser = macho.Parser.init(data);

    // Let's read the magic number from the data
    const magic = try parser.parseLiteral(u32);
    if (magic != 0xfeedfacf) {
        return error.BadMagic;
    }

    std.debug.print("0x{x}\n", .{magic});
}

We can compile our program by running

$ zig build

If it compiles without error (which it should), the macho executable can be found at zig-out/bin/macho:

$ zig-out/bin/macho
Missing required argument: FILENAME
error: MissingArgument
/Users/greg/src/gpanders.com/macho/src/main.zig:13:9: 0x104f6bb9f in main (macho)
        return error.MissingArgument;
        ^

As expected, we get a MissingArgument error since we did not supply a required argument. Let’s give it our exit binary:

$ zig-out/bin/macho ../exit
0xfeedfacf

Hey that’s the magic number! It looks like we are successfully able to parse integers. Before we move on, let’s see what happens if we give macho a non Mach-O object file:

$ ./zig-out/bin/macho build.zig
error: BadMagic
/Users/greg/src/gpanders.com/macho/src/main.zig:38:9: 0x1003d7de7 in main (macho)
        return error.BadMagic;
        ^

Good, as expected we get a BadMagic error.

Inviting friends to the party

One of the great things about using Zig is how well it interacts with C headers and libraries. This will come in handy as we flesh out our parser, because we are going to need lots of enums. But the enum values are already defined for us in /usr/include/mach-o/loader.h (and a few other places). So rather than redo all that work ourselves, we can simply import the existing C header files and reuse those definitions.

First, we need to tell Zig where to look for these header files. In build.zig there is a block of lines that looks like this:

const exe = b.addExecutable("macho", "src/main.zig");
exe.setTarget(target);
exe.setBuildMode(mode);
exe.install();

Just above the line exe.install(), add

exe.addIncludeDir("/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/");

Now, back in src/macho.zig, we can include the C headers:

/// src/macho.zig

const std = @import("std");

// NEW!
const loader = @cImport(@cInclude("mach-o/loader.h"));
const machine = @cImport(@cInclude("mach/machine.h"));

Zig will translate all of the C code found in those two headers and their declarations can be accessed under each respective namespace.

For example, in /usr/include/mach-o/loader.h we find the line

#define	MH_MAGIC	0xfeedface	/* the mach magic number */

We can access this value from Zig using

loader.MH_MAGIC

Let’s leverage this to create some data structures.

First, we’ll create a Zig version of the mach_header_64 struct. This will allow us to use Zig’s much better type system.

/// src/macho.zig

pub const MachHeader64 = struct {
    magic: u32,
    cputype: CpuType,
    cpusubtype: u32,
    filetype: Filetype,
    ncmds: u32,
    sizeofcmds: u32,
    flags: Flags,
    reserved: u32 = undefined,
};

Now we need to define the CpuType and Filetype enums (the enum values of cpusubtype depend on the value of cputype. For simplicity, we’ll just parse this as a raw number rather than defining an actual enum).

This part is a bit tedious. We simply need to copy the #defines from loader.h into our Zig file and transform them into valid Zig syntax. This is no match for some :s Vim-fu, but if you’re following along, feel free to simply copy and paste these definitions:

/// src/macho.zig

const Filetype = enum(u32) {
    object = loader.MH_OBJECT,
    execute = loader.MH_EXECUTE,
    fvmlib = loader.MH_FVMLIB,
    core = loader.MH_CORE,
    preload = loader.MH_PRELOAD,
    dylib = loader.MH_DYLIB,
    dylinker = loader.MH_DYLINKER,
    bundle = loader.MH_BUNDLE,
    dylib_stub = loader.MH_DYLIB_STUB,
    dsym = loader.MH_DSYM,
    kext_bundle = loader.MH_KEXT_BUNDLE,
    fileset = loader.MH_FILESET,
};

const CpuType = enum(u32) {
    // @bitCast here is necessary to convert CPU_TYPE_ANY (-1) to unsigned int
    // (all 1's)
    any = @bitCast(u32, machine.CPU_TYPE_ANY),
    vax = machine.CPU_TYPE_VAX,
    mc680x0 = machine.CPU_TYPE_MC680x0,
    x86 = machine.CPU_TYPE_X86,
    x86_64 = machine.CPU_TYPE_X86 | machine.CPU_ARCH_ABI64,
    mc98000 = machine.CPU_TYPE_MC98000,
    hppa = machine.CPU_TYPE_HPPA,
    arm = machine.CPU_TYPE_ARM,
    arm64 = machine.CPU_TYPE_ARM | machine.CPU_ARCH_ABI64,
    arm64_32 = machine.CPU_TYPE_ARM | machine.CPU_ARCH_ABI64_32,
    mc88000 = machine.CPU_TYPE_MC88000,
    sparc = machine.CPU_TYPE_SPARC,
    i860 = machine.CPU_TYPE_I860,
    powerpc = machine.CPU_TYPE_POWERPC,
    powerpc64 = machine.CPU_TYPE_POWERPC | machine.CPU_ARCH_ABI64,
};

const Flags = packed struct {
    noundefs: bool,
    incrlink: bool,
    dyldlink: bool,
    bindatload: bool,
    prebound: bool,
    split_segs: bool,
    lazy_init: bool,
    twolevel: bool,
    force_flat: bool,
    nomultidefs: bool,
    nofixprebinding: bool,
    prebindable: bool,
    allmodsbound: bool,
    subsections_via_symbols: bool,
    canonical: bool,
    weak_defines: bool,
    binds_to_weak: bool,
    allow_stack_execution: bool,
    root_safe: bool,
    setuid_safe: bool,
    no_reexported_dylibs: bool,
    pie: bool,
    dead_strippable_dylib: bool,
    has_tlv_descriptors: bool,
    no_heap_execution: bool,
    app_extension_safe: bool,
    nlist_outofsync_with_dyldinfo: bool,
    sim_support: bool,
    dylib_in_cache: bool,
    _: u3, // pad to 32 bits
};

The enum(u32) syntax above means that we want those enums to be represented using a u32. The packed keyword on the Flags struct creates a bitfield: each bool field in this struct will use exactly a single bit. The _: u3 at the end pads our struct to a full 32 bits.

Note that we are able to use the definitions from the C header files to define our enums. That means we don’t need to know or care what those values are.

Now, let’s add another function to our parser to parse enum values.

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    ...


    /// Functions
    pub fn init(...) { ... }

    pub fn parseLiteral(...) { ... }

    pub fn parseEnum(self: *Parser, comptime T: type) !T) {

    }
};

Note that this function signature is identical to that of parseLiteral. We could combine these into a single function and use Zig’s type reflection to handle literal values and enums differently. Perhaps we’ll do that later, but for now let’s keep things simple and just use separate functions.

This implementation looks like this:

/// src/macho.zig

pub fn parseEnum(self: *Parser, comptime T: type) !T) {
    const size = @sizeOf(T);
    if (self.data.len < size) {
        return error.NotEnoughBytes;
    }

    const bytes = self.data[0..size];
    const tag = std.mem.bytesToValue(std.meta.Tag(T), bytes);
    const val = std.meta.intToEnum(T, tag) catch |err| {
        std.debug.print("{} is not a valid value for {s}\n", .{
            tag,
            @typeName(T),
        });
        return err;
    };
    self.data = self.data[size..];
    return val;
}

Once again, we check to make sure our byte slice has enough data in it. This time, we convert our bytes into the “tag” type of our enum, which we get using the std.meta.Tag standard library function. For the CpuType enum we defined earlier, this is u32. Once we convert the bytes into the tag type, we can convert the tag value into an enum value using std.meta.intToEnum. This will convert a number like 7 into an enum value like x86. If the enum doesn’t have a value corresponding to the given tag value, intToEnum returns an error. In that case, we print a helpful error message, and return the same error from our function. Note that this should not happen, and indicates a bug in our parser.

It’s also a good idea to encapsulate the parsing logic for our MachHeader64 struct within the struct definition itself. Let’s add a parse method to that struct:

/// src/macho.zig

pub const MachHeader64 = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(parser: *Parser) !MachHeader64 {

    }
};

This parse function will take a Parser object and return a MachHeader64. This is what the implementation looks like:

/// src/macho.zig

pub fn parse(parser: *Parser) !MachHeader64 {
    const magic = try parser.parseLiteral(u32);
    if (magic != loader.MH_MAGIC_64) {
        return error.BadMagic;
    }

    const cputype = try parser.parseEnum(CpuType);
    const cpusubtype = try parser.parseLiteral(u32);
    const filetype = try parser.parseEnum(Filetype);
    const ncmds = try parser.parseLiteral(u32);
    const sizeofcmds = try parser.parseLiteral(u32);
    const flags = try parser.parseLiteral(Flags);
    const reserved = try parser.parseLiteral(u32);

    return MachHeader64{
        .magic = magic,
        .cputype = cputype,
        .cpusubtype = cpusubtype,
        .filetype = filetype,
        .ncmds = ncmds,
        .sizeofcmds = sizeofcmds,
        .flags = flags,
        .reserved = reserved,
    };
}

Hopefully nothing here is surprising. We first parse the magic number and return a BadMagic error if it doesn’t match what we expect. Note that now instead of hardcoding 0xfeedfacf we are using the definition from the loader.h header file.

Next we use our parseEnum and parseLiteral functions to parse each field of the struct and finally place each value into the returned struct value.

Let’s update main.zig:

/// src/main.zig

var parser = macho.Parser.init(data);

const header = try macho.MachHeader64.parse(&parser);
std.debug.print("{}\n", .{header});

If we build and run our program now, we should see a lot more information (note that we are using the shorthand zig build run to both build and run the program at once):

$ zig build run -- ../exit
MachHeader64{ .magic = 4277009103, .cputype = CpuType.arm64, .cpusubtype = 0,
.filetype = Filetype.execute, .ncmds = 16, .sizeofcmds = 744, .flags = Flags{
.noundefs = true, .incrlink = false, .dyldlink = true, .bindatload = false,
.prebound = false, .split_segs = false, .lazy_init = false, .twolevel = true,
.force_flat = false, .nomultidefs = false, .nofixprebinding = false,
.prebindable = false, .allmodsbound = false, .subsections_via_symbols = false,
.canonical = false, .weak_defines = false, .binds_to_weak = false,
.allow_stack_execution = false, . root_safe = false, .setuid_safe = false,
.no_reexported_dylibs = false, .pie = true, .dead_strippable_dylib = false,
.has_tlv_descriptors = false, .no_heap_execution = false, .app_extension_safe
= false, .nlist_outofsync_with_dyldinfo = false, .sim_support = false,
.dylib_in_cache = false, ._ = 0 }, .reserved = 0 }

Woof, that’s not pretty. This “raw” representation is good for debugging, but it’s not very nice to look at.

We can tell print how we want our MachHeader64 struct to be formatted by adding a format() method to MachHeader64.

/// src/macho.zig

pub const MachHeader64 = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(...) { ... }

    pub fn format(value: MachHeader64, comptime fmt: []const u8, options:
    std.fmt.FormatOptions, writer: anytype) !void {
        _ = fmt;
        _ = options;
        try std.fmt.format(writer, "magic: 0x{x}\n", .{value.magic});
        try std.fmt.format(writer, "cputype: {}\n", .{value.cputype});
        try std.fmt.format(writer, "cpusubtype: 0x{x}\n", .{value.cpusubtype});
        try std.fmt.format(writer, "filetype: {}\n", .{value.filetype});
        try std.fmt.format(writer, "ncmds: {d}\n", .{value.ncmds});
        try std.fmt.format(writer, "sizeofcmds: {d}\n", .{value.sizeofcmds});
        try std.fmt.format(writer, "flags: ", .{});
        inline for (std.meta.fields(Flags)) |field| {
            if (field.field_type == bool and @field(value.flags, field.name)) {
                try std.fmt.format(writer, " ", .{});
                inline for (field.name) |c| {
                    try std.fmt.format(writer, "{c}", .{std.ascii.toUpper(c)});
                }
            }
        }
    }
};

The first two lines of this function suppress Zig’s compiler errors for unused function arguments. Next, we simply print each struct field in a manner appropriate for that field’s type. For the bit flags, we want a neat representation that only shows flags that are set. We do this using an inline for loop, which is a loop that is unrolled at compile time. We get a slice of all of the fields of the Flags struct with std.meta.fields, iterate over each of them, and print the field’s name if it’s corresponding bit is set.

Now when we rebuild and run we should see

$ zig build run -- ../exit
magic: 0xfeedfacf
cputype: CpuType.arm64
cpusubtype: 0x0
filetype: Filetype.execute
ncmds: 16
sizeofcmds: 744
flags: NOUNDEFS DYLDLINK TWOLEVEL PIE

Much nicer. This gives us a much better view into the data contained within our Mach-O file.

We see our old friend feedfacf, as well as a few things that we already know, such as the CPU type (arm64) and the filetype (execute, indicating this file is an executable). Following that we see that this Mach-O has 16 load commands which take up 744 total bytes. Finally, this Mach-O file has the NOUNDEFS, DYLDLINK, TWOLEVEL, and PIE flags set:

#define	MH_NOUNDEFS	0x1		/* the object file has no undefined
					   references */
#define MH_DYLDLINK	0x4		/* the object file is input for the
					   dynamic linker and can't be staticly
					   link edited again */
#define MH_TWOLEVEL	0x80		/* the image is using two-level name
					   space bindings */
#define	MH_PIE 0x200000			/* When this bit is set, the OS will
					   load the main executable at a
					   random address.  Only used in
					   MH_EXECUTE filetypes. */

Before we move on to parsing the load commands, let’s make one more improvement to our Parser struct. We added a parse method to the MachHeader64 struct which let us encapsulate the parsing logic for that data type. We want to extend this to all of the structs that we expect to parse, and we want to be able to easily dispatch the right method from our Parser struct.

To do this, we will use more comptime polymorphism. Let’s add a parseStruct method to our Parser:

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn init(...) { ... }
    pub fn parseLiteral(...) { ... }
    pub fn parseEnum(...) { ... }

    pub fn parseStruct(self: *Parser, comptime T: type) !T {

    }
};

We’ll handle this by checking if the type T has a parse method and, if so, simply call that:

/// src/macho.zig

pub fn parseStruct(self: *Parser, comptime T: type) !T {
    if (comptime !std.meta.trait.hasFn("parse")(T)) {
        @compileError(@typeName(T) ++ " does not have a parse() method");
    }

    const val = try T.parse(self);
    return val;
}

Note that in this function we are not checking to make sure our byte slice has enough bytes, nor are we updating self.data. That is because both of these things will be done by the struct’s own parse function.

Notice that parseStruct has the exact same signature as both parseLiteral and parseEnum. Perhaps this is a good type to consolidate all of these into a single parse function.

/// src/macho.zig

pub const Parser = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn init(...) { ... }

    pub fn parse(self: *Parser, comptime T: type) !T {
        if (comptime std.meta.trait.hasFn("parse")(T)) {
            return try T.parse(self);
        }

        const size = @sizeOf(T);
        if (self.data.len < size) {
            return error.NotEnoughBytes;
        }

        const bytes = self.data[0..size];

        const val = switch (@typeInfo(T)) {
            .Enum => |info| blk: {
                const tag = std.mem.bytesToValue(info.tag_type, bytes);
                break :blk std.meta.intToEnum(T, tag) catch |err| {
                    std.debug.print("{} is not a valid value for {s}\n", .{
                        tag,
                        @typeName(T),
                    });
                    return err;
                };
            },
            .Int, .Struct => std.mem.bytesToValue(T, bytes),
            else => unreachable,
        };

        self.data = self.data[size..];
        return val;
    }
};

Here we see a few more Zig features. First, we use a comptime expression in an if statement to check whether or not our T type has a parse method.

Next, we use a switch on @typeInfo(T), which is a tagged union. The switch branches are the different tags. The captured value (|info|) in the .Enum branch is the payload; in this case, the actual type info of the enum. This allows us to replace the call to std.meta.Tag with simply info.tag_type.

We also make use of Zig’s labeled break to return a value from a block expression. The blk: label gives a name to the succeeding block, and break :blk ... uses the following value as the value of the block itself. As before, if std.meta.intToEnum encounters an error, the entire function returns an error. Otherwise, we remove the bytes from our byte slice and return the parsed value.

Be sure to update the parse function in MachHeader64 to use this new variant as well:

/// src/macho.zig

const MachHeader64 = struct {
    /// Field definitions
    ...

    /// Functions
    pub fn parse(parser: *Parser) !MachHeader64 {
        const magic = try parser.parse(u32);
        if (magic != loader.MH_MAGIC_64) {
            return error.BadMagic;
        }

        const cputype = try parser.parse(CpuType);
        const cpusubtype = try parser.parse(u32);
        const filetype = try parser.parse(Filetype);
        const ncmds = try parser.parse(u32);
        const sizeofcmds = try parser.parse(u32);
        const flags = try parser.parse(Flags);
        const reserved = try parser.parse(u32);

        return MachHeader64{
            .magic = magic,
            .cputype = cputype,
            .cpusubtype = cpusubtype,
            .filetype = filetype,
            .ncmds = ncmds,
            .sizeofcmds = sizeofcmds,
            .flags = flags,
            .reserved = reserved,
        };
    }
};

With this abstraction in place, we can update main.zig again:

/// src/main.zig

var parser = macho.Parser.init(data);

const header = try parser.parse(macho.MachHeader64);
std.debug.print("{}\n", .{header});

This will come in handy as we move on to load commands and begin parsing a larger variety of data structures.

Let’s stop here for now. In part 3, we’ll parse the load commands and see what else lies in store in Mach-O.

Full disclosure: I first tried writing the parser in Rust. However, I found the experience pretty frustrating. I don’t want to put all the blame on Rust, as it’s extremely likely that I just wasn’t approaching the task in an idiomatic way, but it felt like the language was fighting me at every turn (and I don’t mean the borrow checker – that part I have no trouble with). After battling to get something working for a couple of days, I finally gave up and did it in Zig and had something working in under an hour. ↩︎

Exploring Mach-O, Part 1

Sun, 16 Jan 2022 19:13:49 -0700

This is part 1 of a 4 part series exploring the structure of the Mach-O file format. Here are links to part 2, part 3, and part 4.

I recently read a great article from Amos over at fasterthanli.me that explored the ELF format for Linux executables. Digging into these kinds of topics in a deep and thorough way has always been super interesting to me, not to mention educational. So, I thought I’d take a stab at doing something similar for Mach-O, the object file format used by macOS.

Before going on this journey I didn’t know anything about Mach-O except that, well, it was the object file format used by macOS. Right now, there’s still a lot I don’t know about Mach-O or the internal workings of macOS, but it’s definitely fair to say that I know it a bit better than I did before!

Feel free to following along if you’d like. The high-level outline of our journey will be:

Create a minimal Mach-O executable
Create a DIY parser (in Zig!) to read Mach-O files
???
Profit

Getting started

The first thing we’ll need is a Mach-O object file. My filesystem is filled with these of course; every binary file on a macOS system is encoded as Mach-O. But for learning purposes that won’t do: we need to do it ourselves.

The smallest possible executable simply calls the exit syscall. If we were on Linux on an x86 machine, this would just be

xor rdi, rdi    ; set rdi to 0 (exit code)
mov rax, $60    ; set rax to 60 (exit syscall number)
syscall         ; do the syscall!

But we’re not on Linux, or on x86! So we need to figure out how to do this in ARM64.

ARM64 doesn’t have the same cryptically named registers as x86, instead using boring old names like x0 and x1 for the first and second function arguments, respectively. ARM64 shares the mov instruction with x86, so we know the first line of our assembly will be

mov x0, #0

We want to call the exit syscall with argument 0 to exit the program cleanly. We could also exit with an error code like 42 to prove to ourselves that our program is doing what we expect:

mov x0, #42

Now we need to make the exit syscall. How do we do that? Well according to the handy ARM developer documentation, we want the svc instruction. However, we also need to know the syscall number.

We can cheat a little bit here and just copy macOS’s homework. The system library located at /usr/lib/system/libsystem_kernel.dylib is part of macOS’s libSystem.dylib, the libc implementation. What does that library have to say about svc?

$ objdump -d /usr/lib/system/libsystem_kernel.dylib | grep -B 2 svc | head -n20
_issetugid:
     e0c:	f0 28 80 d2	mov	x16, #327
     e10:	01 10 00 d4	svc	#0x80
--
__kernelrpc_mach_vm_allocate_trap:
     f64:	30 01 80 92	mov	x16, #-10
     f68:	01 10 00 d4	svc	#0x80
--
__kernelrpc_mach_vm_purgable_control_trap:
     f70:	50 01 80 92	mov	x16, #-11
     f74:	01 10 00 d4	svc	#0x80
--
__kernelrpc_mach_vm_deallocate_trap:
     f7c:	70 01 80 92	mov	x16, #-12
     f80:	01 10 00 d4	svc	#0x80
--
_task_dyld_process_info_notify_get:
     f88:	90 01 80 92	mov	x16, #-13
     f8c:	01 10 00 d4	svc	#0x80
--

Well that’s a good start. Disassembling libsystem_kernel.dylib reveals quite a few svc calls, all of which are present within procedures that look suspiciously like system calls… So it looks like to make a syscall, we move the syscall number into register x16 and then use the svc instruction with an immediate value of #0x80.

We need to know the syscall number though. Maybe we can find exit in there?

$ objdump -d /usr/lib/system/libsystem_kernel.dylib | grep -B 2 svc | grep -A 2 _exit
___exit:
    7b34:	30 00 80 d2	mov	x16, #1
    7b38:	01 10 00 d4	svc	#0x80

Bingo! So it looks like the exit syscall number is #1. Can we confirm this in a more “official” way, perhaps through a definition in a header file or something?

It turns out we can, by taking a peek in /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/syscall.h¹ ². Looking in that header file, we find a list of all of the syscall numbers on macOS:

#define	SYS_syscall        0
#define	SYS_exit           1
#define	SYS_fork           2
#define	SYS_read           3
#define	SYS_write          4
#define	SYS_open           5
#define	SYS_close          6
#define	SYS_wait4          7

And whaddya know, there’s SYS_exit sitting nice and pretty next to 1.

We now know enough to create our minimal Mach-O program. Let’s put it all together:

; exit.asm
_main:
        mov x0, #42     ; exit code
        mov x16, #1     ; syscall number for exit
        svc #0x80       ; do the syscall!

Let’s assemble it!

$ as exit.asm -o exit.o

Technically, exit.o (the object file) is itself a Mach-O file. So we could just stop here and move on with the parsing. But we should probably make sure our tiny program works, no? So now let’s link the object file into a full blown executable.

$ ld exit.o -o exit
ld: warning: arm64 function not 4-byte aligned: ltmp0 from exit.o
Undefined symbols for architecture arm64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture arm64

Uh oh. That’s not pretty. It turns out, there are a few more things our little ASM program needs. First, we need to ensure that the start address of the _main function is 4-byte aligned. We can do this by adding a line

.align 4

just before the start of the _main function. We also need to tell the assembler to make _main visible to the linker by adding

.global _main

The final version should look like this:

; exit.asm
.global _main
.align 4
_main:
        mov x0, #42     ; exit code
        mov x16, #1     ; syscall number for exit
        svc #0x80       ; do the syscall!

Ok let’s try linking again!

$ ld exit.o -o exit
ld: dynamic main executables must link with libSystem.dylib for architecture arm64

Blast! This error message is pretty self-explanatory: we need to link with libSystem.dylib. Ok, no problem, we’ll just add -lSystem to the ld invocation.

$ ld exit.o -lSystem -o exit
ld: library not found for -lSystem

Eh? One might expect libSystem.dylib to be on the default library search path, but apparently as of macOS 11, one would be wrong. So we need to explicitly add the oh-God-why-is-it-so-long library path to the command line:

$ ld exit.o -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/ -lSystem -o exit

Finally, no error message, which means linking was successful! If we run our program and check the return code, we should see 42:

$ ./exit
$ echo $?
42

Cool! If we disassemble our executable with objdump we should see the very same commands we just wrote by hand:

$ objdump -d exit

exit:   file format mach-o arm64


Disassembly of section __TEXT,__text:

0000000100003fa0 <_main>:
100003fa0: 40 05 80 d2  mov     x0, #42
100003fa4: 30 00 80 d2  mov     x16, #1
100003fa8: 01 10 00 d4  svc     #0x80

No surprises here. Notice that objdump mentioned that this is the disassembly for “section __TEXT,__text”. We don’t know what that means yet, but we’ll find out soon enough.

Let’s get macho

Now that we have a tiny executable we can start investigating the Mach-O format in more detail. It just so happens that archive.org has a link to the Mac OS X ABI Mach-O File Format Reference. Surprisingly, I wasn’t able to find anything as in-depth as this from Apple directly. This document will guide us on our way to parsing our little Mach-O file. It is a bit old, and there are a few things that are either missing or incorrect. We will also reference the header files found under /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/mach-o.

Reading through the reference document, we see that Mach-O files have three major “regions”:

A header
A list of “load commands”
Segments

The header is a simple 32-byte structure that looks like this:

struct mach_header_64 {
   uint32_t magic;
   cpu_type_t cputype;
   cpu_subtype_t cpusubtype;
   uint32_t filetype;
   uint32_t ncmds;
   uint32_t sizeofcmds;
   uint32_t flags;
   uint32_t reserved;
};

Note that this is the header for 64 bit executables. 32-bit Mach-O files use a slightly different header, but we’re going to assume 64-bit for the rest of our journey.

The first four bytes of every Mach-O file is the “magic number”, just like in ELF files. ELF uses the magic number 0x7F 0x45 0x4C 0x46 (which is just 0x7F ELF). According to the Mach-O File Format Reference, the magic number for 32-bit Mach-O files is defined as the constant MH_MAGIC. Where is this constant defined? According to the reference, it’s in /usr/include/mach-o/loader.h. Let’s see what it is:

$ printf 'MH_MAGIC' | cc -include 'mach-o/loader.h' -E - | tail -n1
0xfeedface

I… uh… ok. Yes, the magic number for Mach-O files is, in fact, feedface. I’ll be honest, I got quite a kick out of this.

64-bit Mach-O files use the constant MH_MAGIC_64, which is just MH_MAGIC + 1, i.e. 0xfeedfacf. Not as funny.

Let’s check our exit program and see for ourselves:

$ # -l 4 = read 4 bytes, -e = little endian
$ xxd -l 4 -e ./exit
00000000: feedfacf                             ....

Yup. There it is. feedfacf.

After the magic number are a few enums: cpu_type_t, cpu_subtype_t, and filetype. We could, at this point, continue to poke around our program using xxd (or your hex editor of choice) and compare the raw byte values with the definitions of these enums in the mach-o/loader.h header file. But that’s a bit tedious. Let’s write some code.

In part 2, we’ll start on our own DIY parser to read through our Mach-O file.

Requires the Command Line Tools for Xcode. ↩︎
By the way, since this path is painfully long to both read and write, for the remainder of this article I’m going to pretend there is an imaginary symlink from /Library/Developer/CommandLineTools/SDKs/usr/include to /usr/include. So any references to /usr/include/* are actually under the full path. ↩︎

The definitive guide to using tmux-256color on macOS

Wed, 14 Apr 2021 00:00:00 +0000

UPDATE: macOS 14 (Sonoma) updated its bundled ncurses version to 6.0. This version ships with a tmux-256color entry by default, so the workaround below is no longer needed.

macOS ships with an old and very out-of-date version of ncurses (5.7). This version does not contain a terminfo entry for tmux-256color, the preferred TERM value when using tmux.

If you installed tmux via Homebrew or MacPorts, then you will also have a newer version of ncurses (6.X) present on your system installed by your package manager. This is the version of ncurses that tmux itself was built against, which is why you can use

set -s default-terminal tmux-256color

in your .tmux.conf without any errors. However, as soon as you try to use any application linked against macOS’s version of ncurses (such as man or ls -G), you will notice that things no longer work.

The solution is to get the tmux-256color terminfo entry from the newer version of ncurses into macOS’s terminfo database. First, create a copy of the current tmux-256color terminfo entry from the version of curses installed by your package manager:

# MacPorts
$ /opt/local/bin/infocmp -x tmux-256color > ~/tmux-256color.src

# Homebrew
$ /usr/local/opt/ncurses/bin/infocmp -x tmux-256color > ~/tmux-256color.src

Now, modify the tmux-256color.src file to change the pairs value from 65536 to 32767. This must be done because of a bug in ncurses 5.7 that interprets pairs#65536 as pairs#0¹.

Before:

pairs#0x10000

pairs#65536

After:

pairs#32767

Prior to macOS Catalina, it was possible to install this new terminfo entry directly into the system database /usr/share/terminfo. However, as of Catalina we can no longer do this since the root volume (of which /usr/share/terminfo is a part) is read-only. There are, apparently, workarounds to make the root volume writable, so if you want to go that route you can. Instead, we can install the terminfo entry to a local database under $HOME:

$ /usr/bin/tic -x -o $HOME/.local/share/terminfo tmux-256color.src

Be sure to explicitly use /usr/bin/tic to ensure that the entry is compiled with the builtin ncurses. This will install the terminfo database entry under $HOME/.local/share/terminfo. ncurses uses the TERMINFO_DIRS environment variable to specify a location to search for terminfo entries when not found in the primary database (see man 5 terminfo). We can take advantage of this to have the macOS 5.7 version of ncurses search in a directory of our choice when it doesn’t find tmux-256color under /usr/share/terminfo, while the 6.X version provided by the package manager will still use its own version².

Simply set the TERMINFO_DIRS environment variable in your shell startup file (e.g. .bashrc, .zshrc, etc.) to use that directory (note that we append this value to the existing value, if any, similar to how we append values to $PATH):

$ export TERMINFO_DIRS=$TERMINFO_DIRS:$HOME/.local/share/terminfo

After that, you should be able to correctly use tmux-256color as your TERM variable. You can check that everything is working by running a few programs that make use of ncurses and terminal colors, such as htop, man, ls -G, etc.

See Thomas Dickey’s comment here. ↩︎
An earlier version of this article installed the terminfo entry into $HOME/.terminfo, which is a suboptimal solution as even the updated 6.X version of ncurses would use the terminfo entry with reduced colors. The use of TERMINFO_DIRS was suggested to me by Carlo Cabrera, and provides the best of both worlds. Thanks Carlo for the tip! ↩︎

The Zettelkasten

Mon, 25 May 2020 15:43:50 -0600

I have always been a bit of a nerd about note taking and knowledge management systems. I read David Allen’s Getting Things Done with relish, and I’ve obsessively experimented with every iteration of note taking apps and tools mankind has created. Trello, Todolist, Evernote, Bullet Journal, you name it.

I think my obsession in this area is mostly driven by the combination of my love of useless trivia and my immense capacity for forgetfulness, a perfect recipe if ever there was one for developing the habit of writing things down.

Of course, being an obsessive optimizer means that I can’t just settle for any solution; I have to find the BEST solution. The result is that I’ve drifted from app to app, from method to method, ever searching for the metaphorical glass slipper that perfectly fits my needs.

Not long ago, while reading a blog post by another information management nerd, I stumbled upon something called a “Zettelkasten”. At first, I was put off: I felt there was already enough diversity in the note taking ecosystem without delving into other languages. Once I’d gone down that rabbit hole, who knows where it would end up?

However, my curiosity was piqued, and I dipped my toe in the water. I have since become hooked, and I now use a Zettelkasten as my primary note taking and knowledge management method. In the rest of this article, I’ll explain exactly what a Zettelkasten is and why it meets my needs.

An important disclosure is necessary before we continue: if I’ve learned anything in my quest to find the perfect knowledge management system, it’s in that there is no one-size-fits-all approach. The reason there are 7 billion different note taking apps and methods is because there are 7 billion different people on Earth. One of the reasons I like the method I’m about to outline below is precisely because it’s flexible and moldable to my own needs, but just because it works for me doesn’t mean it’ll work for you.

What is a Zettelkasten?

Quoting from the blog post above (emphasis mine):

[A] precise definition of Zettelkasten is likely to be counterproductive. Short of saying it is a loosely defined method of constructing an archive of notes. An archive built upon layers of nodes and connections. If you want to know more, however, Christian and Sascha have a growing archive of their own at the Zettelkasten blog. In case you don’t already know how philosophical note taking can be, you have been warned.

Zettelkasten.de is the canonical home of everything Zettelkasten-related. It’s written and maintained by two guys (the aforementioned Christian and Sascha) who are passionate about efficient note taking processes (did you ever imagine that such people existed? Alas, I am apparently one of them). Put simply, a Zettelkasten is like an external brain, a flattened structure of information that connects to itself like a web, linking together all of the information it contains in ways that might not be apparent at first, creating insights for synthesizing new knowledge.

In German Zettel means “note” and a Kästen is a box: a Zettelkästen is literally translated as “note box”. Its inventor, Niklas Luhmann, literally kept a box of index card-style notes. Each note had a unique identifier that could be referenced by other notes, creating an interconnected web of knowledge. Modern Zettelkastens can of course still be created this way, but most people find it easier and more efficient to use digital versions.

One of the defining characteristics of a Zettelkasten over other forms of knowledge management is its use of unique identifiers. In a Zettelkasten, each Zettel has a unique ID that other Zettels can refer to. Links are created between Zettels by placing IDs within notes, and when one note references another note’s ID, that note can itself be updated with a link to the linking note, creating a backlink.

Over time, a Zettelkasten grows in an organic fashion to truly become an external brain. As you enter more and more information into your Zettelkasten, you will find that links develop between notes that you may not have ever have anticipated. These connections can then help you synthesize new information or succinctly put together complex ideas in a simple way.

Another defining characteristic of the Zettelkasten is its eschewing of hierarchy and categories. Instead, Zettels are tagged. This might seem like a minor distinction, but it pays off in one important way: it eliminates any friction when creating a new note caused by asking oneself to which predefined category that note ought to belong. Instead, the new Zettel is simply created in a flat structure and any tags that seem appropriate to the content can be entered. Then, when you want to query your Zettelkasten for anything you’ve written about bumblebees you simply search for notes tagged with “bumblebee” or that contain the phrase “bumblebee”.

Another appealing feature of the Zettelkasten is that it can (and should!) work perfectly well in plain text. In fact, the folks over at Zettelkasten.de insist on this, and I agree. Anything that you want to last needs to be in plain text: proprietary or application-specific formats are no good and interfere with your knowledge’s ability to be fluid and flexible. If you put everything in Evernote, you’re stuck in Evernote unless you want to go through the arduous process of exporting your data into a format that will almost certainly come out funky or malformed.

The Zettelkasten.de guys use MultiMarkdown as their plain text format of choice, but I just use Pandoc’s Markdown syntax. Not that it really matters, since the whole point of using plain text is that it’s perfectly readable as plain text. Write them in any format you want, as long as it’s consistent and easy to search.

All of these attributes mean that creating and managing a Zettelkasten is low friction, low maintenance, and highly portable. Individual Zettels are meant to be (generally speaking) small and atomic. You don’t need to worry about if the note is in the right place, or what’s written in it. Just write it! If you think of something else, modify the existing note if it’s relevant or just create a new note and link back to the original. In this way, the Zettelkasten grows organically over time along with its creator.

The aforementioned traits also mean that implementing your own Zettelkasten tool is quite trivial. I did this myself with wk (which I’ll write about soon), but you can also use something like The Archive or nvALT.

Examples

Ok, you’re a visual learner. I get it, I am too. Here’s an example of what one of my own Zettels looks like:

---
title: Alexander von Humboldt
date:  December 13, 2019
tags:  romanticism, nature, science, ecology
...

Alexander von Humboldt was among the first in the post-Enlightenment modern era
to think of nature as something to be enjoyed, commune with, and treasure
instead of simply to exploit and command. One of the last polymaths of the
Enlightenment, he re-imagined science's relationship with nature, insisting
that nature was not simply mechanistic but rather highly complex and
interwoven. His discipline would later come to be known as ecology
[[20191213143234]].

### Influenced

- George Perkins Marsh [[20191213092520]]
- Ernst Haeckel [[20191213151234]]
- John Muir
- Charles Darwin
- Goethe

### Backlinks

- [[20191213092520]]
- [[20191213151234]]

Let’s go over the structure here.

At the top of the note is the metadata section. This is an extension of Pandoc’s Markdown format. It specifies a title, date, and tags. It’s fairly self-explanatory.

After the metadata is the note body itself. Note the use of the [[20191213143234]] style links. These are references to other notes. Also take note of the ## Backlinks section at the bottom: this is a list of other notes that link to this note in turn.

That’s it. In other notes, I might also have a citation section where I include citations of things I reference in the note itself. I use Zotero to manage references and citations, and copying a citation from Zotero into a new note is a simple copy-paste operation.

I interact with notes in my Zettelkasten with the aforementioned wk tool. Like I mentioned, I’ll write about this in more detail later, but briefly, it’s a command line tool with commands like “open”, “new”, and “search”. For example, if I just finished reading an interesting blog post about the history of bear migration in North America, I would simply type the following in my terminal:

wk new 'Bear migration in North America'

This would open a newly created note prepopulated with the relevant metadata in my editor (Vim) and allow me to quickly start writing. In the tags: field, I might put bears or migration. When I’m done, I just save and close the note and go about my day. The whole process is very fast and very low friction.

Conclusion

I have been growing my own Zettelkasten for about 6 months now and have, amazingly, been able to get the process to stick, which is more than I can say for most other processes I’ve tried.

It’s worth repeating my disclosure at the beginning of this post: the system I’ve outlined above works well for me. It might seem byzantine, awkward, or bizarre to you, and that’s okay. The perfect note taking system is the one that will actually get you to write notes and use the notes you’ve written down. In that sense, the Zettelkasten method is, for me, a huge success.

Introducing ijq, an Interactive jq Tool

Sat, 23 May 2020 19:07:09 -0600

If you’re unfamiliar with the tool jq, it describes itself as the “sed for JSON”. It’s a powerful command line tool that lets you do complex parsing and manipulation of JSON documents and is extremely useful for testing APIs or analysing JSON datasets.

One challenge with jq, however, is finding just the right filter. The query language that jq uses is not always obvious, so typically you’ll spend a bit of time getting it just right to select the data you need.

jq play solves this problem by providing an interactive jq environment that allows you to see the filtered results of your query in real time. vim-jqplay is similar, but runs inside of Vim. I thought this was a pretty neat idea, so I thought I’d take a stab at making something similar that works not in Vim but directly on the command line. I call it ijq, or “interactive jq”.

Mostly this was just an excuse to get some more practice with Go (which is what I used to write ijq). But I do think this will actually be quite useful. It mostly works as a drop-in jq replacement, since you can use it to output the resulting JSON object to another file or process.

Here’s a quick video:

You can get ijq over on sourcehut.

Managing Dotfiles With Git

Fri, 01 May 2020 00:00:00 +0000

Managing so-called “dotfiles” (config files starting with a . in your home directory) in git is a fairly common practice. There are many write-ups on this topic already, but I wanted to share my own workflow as well as some potentially useful tips and tricks.

For those unaware, dotfile management basically consists of adding files such as your .vimrc, .tmux.conf or other files in a git repository so that you can both track changes in these files over time as well as share them across multiple computers. If you have a home laptop and a work computer, you can ensure you have the same setup on both by keeping your files in a git repository that is synchronized on both machines.

There are a few common strategies for doing this. Perhaps the simplest (and the method I used for quite a while) is to keep all of the files in a separate directory and symlink them into your home directory. For example, you might have a directory called .dotfiles in your home directory that contains your .vimrc file and symlink it into your home directory using

$ ln -s ~/.dotfiles/.vimrc ~/.vimrc

The advantage of this approach is that it allows you to use any file naming and organizational scheme you want. You can keep all files related to Vim in a vim directory and all files related to a program called foo in a separate foo directory. As long as the symlink is in the right place with the right name, it doesn’t matter where it is in your git repo.

A second approach, which I now use, is to use your home directory as the git repository and manage the files directly. Atlassian has a good write up on this method. My setup is almost identical to the one explained in that article, with the exception that I clone my git repo to .dotfiles instead of .cfg and instead of using a separate cfg alias to manage my dotfiles repo, I simply created a wrapper shell function that behaves differently if I am in my home directory:

git() {
	if [ "$PWD" = "$HOME" ]; then
		command git --git-dir="$HOME/.dotfiles" --work-tree="$HOME" "$@"
	else
		command git "$@"
	fi
}

What I like most about this method is how easy it is to bootstrap a new machine with your dotfiles repo:

$ git clone --bare git@codeberg.org:gpanders/dotfiles .dotfiles
$ git --git-dir=$HOME/.dotfiles --work-tree=$HOME checkout -f

You’ll also probably want to set the status.showUntrackedFiles config option to no so that git status doesn’t list all of the untracked files in your home directory.

I use many different computers on a day-to-day basis. I have a personal laptop, a work laptop, and a work desktop, as well as multiple servers that I often SSH into. In my dotfiles repository, I keep all of the files that I want to use across all of my machines in the master branch, and on each individual machine I create a new branch that contains commits that are specific to that machine. For example, on my work computers I add my work email into my mutt and mbsync configurations, which I do not want on my other machines. Changes that I want to propagate across all of my different computers get committed onto master and then I simply rebase my machine-specific branch onto master.

What is the best method for adding a commit to master while on the machine-specific branch? The simplest and naïve approach is to simply checkout the master branch, add and commit the changes, checkout the machine branch, and then rebase:

$ git checkout master
$ git add file1 file2
$ git commit -m "Updated file1 and file2"
$ git checkout machine
$ git rebase master

This is not ideal. Not only is it rather cumbersome, it also can introduce a lot of friction if there are conflicts between the modified files in your machine-specific branch and master, requiring you to first do a git stash before git checkout, followed by a git stash pop and resolving any conflicts that occur. Instead, we can create the commit directly on our machine-specific branch, and then just copy it on to master after the fact:

$ git add file1 file2
$ git commit -m "Updated file1 and file2"
$ git checkout master
$ git cherry-pick machine
$ git checkout machine
$ git rebase master

This can also be nice for creating multiple commits all at once:

$ git add file1 file2
$ git commit -m "Updated file1 and file2"
$ git add file3
$ git commit -m "Updated file3"
$ git checkout master
$ git cherry-pick machine~2..machine
$ git checkout machine
$ git rebase master

This is what I did for a long time. I even smashed it all together into one single, long command that was conveniently saved in my shell history so that I didn’t have to keep retyping it.

This is still not perfect though. Every time you change branches in your dotfiles repository, the config files for your programs change. This can be annoying or even cause problems for programs that live reload when they detect a change in their config file. Therefore, we would ideally like to be able to commit changes onto master without ever having to leave the machine-specific branch.

Enter the interactive rebase.

In case you’re not aware, the interactive rebase is the single greatest tool git provides. It is extraordinarily powerful, and learning and becoming comfortable with this tool will actually change your life. Check out the chapter in the Git book for more info.

What we’ll do is create our commits on our machine-specific branch as before, then use interactive rebase to move them to the “bottom” of the machine-specific branch, then update the master pointer to point to the last of those commits. This will make more sense in picture form.

Before making any changes:

[ C1 ] <- machine
   |
[ C2 ]
   |
[ C3 ]
   |
[ C4 ] <- master

Now, while still on the machine-specific branch, we create the commits that we want to put on master (represented as commits M1 and M2 in the diagram below):

[ M1 ] <- machine
   |
[ M2 ]
   |
[ C1 ]
   |
[ C2 ]
   |
[ C3 ]
   |
[ C4 ] <- master

Now we run

$ git rebase -i master

which brings up the following in your editor:

pick C3 Commit message for C3
pick C2 Commit message for C2
pick C1 Commit message for C1
pick M2 Update file1 and file2
pick M1 Update file3

We modify this in our editor to move the last two lines (commits M1 and M2) to the top of the stack, and then add an exec line to update the master branch:

pick M2 Update file1 and file2
pick M1 Update file3
exec git branch -f master
pick C3 Commit message for C3
pick C2 Commit message for C2
pick C1 Commit message for C1

Save and close the editor, and done! Now the commits are on master and we never had to change our current branch.

The files in the work tree are still modified during the rebase, however. I haven’t yet figured out a way around this, but if I do I will update this post.

We can of course also push master up to the remote:

$ git push origin master:master

And we can also update master with any changes that were pushed from another machine:

$ git fetch origin master:master
$ git rebase master

This describes my workflow with how I manage my dotfiles. If you’re not used to using rebase, it might seem a little daunting at first. But I assure you that once you get used to it it takes no time at all and becomes just another git operation.

Building Site Search With Hugo

Fri, 17 Apr 2020 13:08:49 -0600

Search is a handy feature you might want on your Hugo static site. Hugo does not have any built-in way to provide a search function, but they do offer some suggestions on how to do it yourself.

The difficulty with implementing a search function is that it usually requires some kind of server side logic, which is obviously missing when using a static site like Hugo. So any search implementation must be done entirely on the client side. One of the options Hugo suggests offers an implementation using jQuery and Fuse.js.

This seemed a bit overkill to me. I did not want to have to add two (rather large) JavaScript libraries to my site simply to provide a simple feature like search. Further, even after trying Fuse.js I was utterly confounded by the search results it returned and found the documentation incomplete and confusing. For example, while testing Fuse I searched my site for “fpga”. As of this writing, there are two posts on this site containing the word “fpga”, so I would expect the search to return those two posts and those two posts only. However, Fuse completely failed to find one of the posts and also returned several false positives! This happens because Fuse uses a fuzzy search algorithm, so the search term “fpga” matches a lot of things besides the term itself, which is not what I wanted.

Fortunately, there is a better way.

Hugo can create a JSON index of your entire site, with keys of your choosing. For example, you can create an index.json file with keys such as title, tags, permalink, date, and contents. You can then use JavaScript to download this index.json file, parse it into a JavaScript object and then search through it (this is also how the Fuse implementation works).

To tell Hugo to create a index.json of your site, add the following to your config.toml file:

[outputs]
  home = ["html", "json"]

Also create the following file at layouts/_default/index.json:

{{- $.Scratch.Add "index" slice -}}
{{- range .Site.RegularPages -}}
    {{- $.Scratch.Add "index" (dict "title" .Title "date" (dateFormat "2006-01-02" .Date) "tags" .Params.tags "contents" .Plain "permalink" .Permalink) -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

When you use hugo to publish your site, you will find a file at public/index.json containing a JSON representation of your whole site.

Once the index.json file is created, we can download it with

var httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = function() {
    if (httpRequest.readyState === 4 && httpRequest.status === 200) {
        const index = JSON.parse(httpRequest.responseText);
        // Search `index` for your search query
    }
};

httpRequest.open("GET", "/index.json");
httpRequest.send();

Then it’s just a matter of searching the index object for our query:

// XMLHttpRequest boilerplate from above
// ...
const index = JSON.parse(httpRequest.responseText);
const keys = ["title", "permalink", "tags", "contents"];
const pattern = new RegExp(query);  // See below for how we get the query
const results = index.filter(item => {
    for (var i = 0; i < keys.length; i++) {
        const key = keys[i];
        if (!item[key]) {
            continue;
        }

        if (key === "tags") {
            if (item[key].some(tag => pattern.test(tag.toLowerCase()))) {
                return true;
            }
        } else if (pattern.test(item[key].toLowerCase())) {
            return true;
        }
    }

    return false;
});
// ...

The items in results are your search results. Now it’s just a matter of adding those results to the DOM to present them to your user.

I added a form to the navigation banner on this site with a single input field that redirects users to gpanders.com/search/ with a single query param q. So a search URL looks like gpanders.com/search/?q=query_phrase. To get the search query from the request URL, I simply use:

const match = window.location.href.match(/\?q=([^&]+)/);
if (!match) {
    return;
}

const query = decodeURIComponent(match[1].replace(/\+/g, " "));
const pattern = new RegExp(query.toLowerCase().replace(/[.*+\-?^${}()|[\]\\]/g, '\\$&'));

We capture the query parameter from the URI and decode it. We then force query to be lower case to make searches case-insensitive and also escape characters that are semantically meaningful in regular expressions. This ensures that the search query entered by the user is interpreted literally and not as a regular expression. If you want search queries to use regular expression syntax, then simply omit the .replace() call above. pattern is then used in the XMLHttpRequest callback we wrote above.

Save this JavaScript file to assets/js/search.js. It needs to be under the assets directory so that we can access it using Hugo’s resources API.

To create the actual search page in Hugo, create the file content/search/index.md with the simple contents:

---
layout: "search"
---

Then create the layout file for this page at layouts/_default/search.html. For example,

{{ with resources.Get "js/search.js" | fingerprint }}
<script async type="text/javascript" src="{{ .Permalink }}" integrity="{{ .Data.Integrity }}" crossorigin="anonymous"></script>
{{ end }}

<h2 class="search-header"></h2>
<ul id="search-results">
</ul>

The way your layout is structured will depend on the theme you are using. You should use the single page layout files from your theme as a starting point and modify it from there.

Then, in your search.js file, fill in the DOM using your search results:

// Generate `results` as in the code block above
// ...
var html = "";
for (var i = 0; i < results.length; i++) {
    html += '<li><a href="' + results[i].permalink + '">' + results[i].title + '</a></li>'
}

document.getElementById("search-results").innerHTML = html;

Et voila! You now have working search on your static Hugo site!

The post above is not meant to be a finely detailed, step-by-step guide, but rather a high-level overview of how I did my implementation. There are a lot of gaps you’ll have to fill in for yourself, but of course, that’s the fun part! Feel free to contact me with any questions or feedback. You can also find the full implementation on sourcehut.

Books for Beginner Economists

Thu, 16 Apr 2020 00:00:00 +0000

I am not an economist by any definition of the word: I haven’t taken a formal economics class since high school. But in the last few years I’ve read a few books that have introduced me to the wonderful, fascinating world of economics, which I now consider one of my favorite subjects.

Two books that provide a good grasp on foundational economics are Naked Economics by Charles Wheelan and Undercover Economist by Tim Harford. Both books offer a “qualitative” introduction to economic concepts: no equations, formulas, or scary math. Both answer fundamental questions about human behavior using tools of economics and provide a framework for answering other questions using economics.

Both of these books, however, discuss economics within the flawed framework of assuming that humans are rational actors who make rational choices. This is a great place to start learning about economics, because to a first approximation this assumption works quite well in evaluating how people make decisions. However, we see some of the predictions and ideas based on this framework begin to break down in the “edge cases” where humans behave in a systematically irrational way.

Enter behavioral economics, the branch of economics that blends together with psychology to study economic choices and decision making in the light of human irrationality. Daniel Kahneman and Amos Tversky did much of the seminal research in this field (Kahneman won a Nobel Prize for his work), and you can read about the psychological underpinnings in their book Thinking Fast and Slow.

However, for an economics perspective I highly recommend Nudge by Richard Thaler and Cass Suntein. Nudge acts as a handbook for decision and policy makers in a world defined by unpredictable and irrational choices. Reading this book will help you understand yourself better, as well as provide valuable insight into how we can create solutions to economic problems that actually work.

Check out these books from your local library or use the links above to buy them on Amazon. If you do read any of these books, I’d love to hear what you think!.

Creating Your Own Git Server

Wed, 18 Mar 2020 00:00:00 +0000

Ever considered setting up and running your very own git server? It’s actually quite easy! In this post, I’ll outline the steps I took to set up my own so that you can give it a try yourself. But first, why might you even want to go through the trouble of setting up your own server?

After all, there are a wide array of excellent and free to use choices out there, such as GitHub, GitLab, and the up-and-coming sourcehut.

One reason is ownership: in today’s world of corporate surveillance, rampant privacy violations, and data breaches, there is something to be said of truly owning your own data. Both git and the web itself were designed and built on principles of decentralization and distribution. Standing up your own server is one way to tap into that heritage.

It’s also just plain fun, at least if you’re into that sort of thing. You get to build something useful and put your name on it. It’s something you control. You get to decide how it works, how it looks, who can access it, and what exists on it.

Setting up a git server is actually relatively straight-forward. Almost all of the heavy lifting is done by git itself, but I will also introduce a few supplementary tools to handle things like access control and HTTP access.

Evaluating the choices

Depending on your needs, you may want to install something like GitLab’s Community Edition which is very easy to set up and provides things like user accounts, issue tracking, and more.

However, if all you need is a simple single-user system where you (and maybe a few other trusted collaborators) are the only one making changes to your source code, GitLab may be a bit overkill. Other options include Gitea or Gogs, both of which offer slimmed down versions of full-scale offerings like GitLab and are designed to run on smaller hardware like a Raspberry Pi.

In this guide, we’ll be using Gitolite to provide and manage SSH access to our git repositories. Gitolite does not offer any kind of HTTP access or a web frontend: for this, we’ll turn to the old-fashioned (but still viable!) gitweb. Finally, we’ll use the built-in git daemon to serve repositories over the git:// protocol.

Preparing your server

You can run your git server on something as simple as a Raspberry Pi or a Digital Ocean droplet. It’s not very demanding, so you don’t need much horsepower.

First, create a git user on your server. This user’s home directory will act as the base location for all of your repositories. I chose /var/lib/git as the home directory, but /home/git or /srv/git are other common choices.

# useradd -r -d /var/lib/git git

Make sure to set the password for the new user:

# passwd git

And make sure you can SSH to the git user on your server:

$ ssh git@yourserver.com

Gitolite

Next, install the excellent Gitolite tool. Gitolite makes managing your server much easier and allows you to do things like user management, access control, triggers, hooks, and more. Copy your SSH public key into your git user’s home directory and then clone Gitolite into the git user’s home directory.

From your home computer (i.e. not your server):

$ scp ~/.ssh/id_rsa.pub git@yourserver:yourname.pub

From your server:

$ su - git
$ git clone https://github.com/sitaramc/gitolite
$ mkdir -p ~/bin
$ gitolite/install -ln ~/bin
$ gitolite setup -pk yourname.pub

Now from your home computer, test your Gitolite installation:

$ git ls-remote git@server:gitolite-admin

You should get something like:

9dd8aab60bac5e54ccf887a87b4f3d35c96b05e4    HEAD
9dd8aab60bac5e54ccf887a87b4f3d35c96b05e4    refs/heads/master

The gitolite-admin repo is where all of the Gitolite configuration takes place. You can clone this to your home computer and push your changes back to your server and Gitolite will automatically re-configure itself. This is also how you add new SSH keys (either for yourself or for other users).

Gitolite also interfaces quite nicely with git-daemon and gitweb. Be sure to check out the thorough documentation on Gitolite’s website.

git daemon

Git includes a daemon subcommand that will listen for incoming connections and serve data over the git:// protocol. This allows you to clone repos from your server using

$ git clone git://yourserver/repo.git

The git:// protocol is the fastest and simplest option for read-only access, and it’s trivial to set up. Assuming your git user’s home directory is /var/lib/git, simply create the following systemd service file (e.g. at /etc/systemd/system/git-daemon.service):

[Unit]
Description=Start Git Daemon

[Service]
ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/var/lib/git/repositories /var/lib/git/repositories
Restart=always
RestartSec=500ms
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=git-daemon
User=git
Group=git

[Install]
WantedBy=multi-user.target

Then simply run systemctl enable --now git-daemon. Make sure port 9418 is open on your firewall.

If you don’t use systemd, you can probably find alternative init scripts for your particular init system in your package manager. Debian, for example, has git-daemon-sysvinit and git-daemon-run if you use SysV or runit, respectively.

The git daemon only serves repositories that have a file called git-daemon-export-ok in their directory root. This allows you to control which repositories you want to be publicly accessible. Gitolite will automatically create this file for you for any repos that are readable by the special daemon user.

Gitweb and HTTP

The most difficult part of setting up a git server is the web frontend and HTTP access. The git documentation has a section on Smart HTTP which is a good starting point, but it still took me quite a while to get everything working. Part of the reason I had some difficulty was because I insisted on running the webserver in a Docker container (so that I can version control the configuration).

By default, your git installation includes the git-http-backend script which allows you to clone over HTTP. If you also want to host a web frontend, then you have to create some rules for your webserver so that it knows when to serve the web page and when to direct traffic to the git-http-backend script.

Here is an example Apache configuration file:

ServerName yourserver.com

DocumentRoot /usr/share/gitweb
<Directory "/usr/share/gitweb">
    DirectoryIndex gitweb.cgi
    Options ExecCGI FollowSymLinks SymLinksIfOwnerMatch
    AddHandler cgi-script cgi
    Require all granted
    SetEnv GITWEB_CONFIG /etc/gitweb.conf
    RewriteEngine On
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_URI} !^/([^/]+/(HEAD|info|objects|git-(upload|receive)-pack).*)$
    RewriteRule ^.* /gitweb.cgi/$0?js=1 [L,QSA,PT]
</Directory>

ScriptAliasMatch \
    "(?x)^/([^./]+)(\.git)?/(HEAD | \
                info/refs | \
                objects/info/[^/]+ | \
                git-(upload|receive)-pack)$" \
    /usr/libexec/git-core/git-http-backend/$1/$3

<Files "git-http-backend">
    SetEnv GIT_PROJECT_ROOT /var/lib/git/repositories
    Require all granted
</Files>

The above configuration allows repos to be cloned with or without the .git extension, but since Gitolite creates all repositories with the .git suffix you’ll have to create non-suffixed symlinks to those directories.

Create and modify the /etc/gitweb.conf file to configure Gitweb.

I run Apache behind a reverse proxy that handles all of the TLS encryption, so the configuration above doesn’t deal with any of that. Setting up TLS for a webserver is out of scope for this guide and if that’s not something you care to do, you can always omit the HTTP/S portion of your server and simply serve via git:// and SSH. Like I said, this part is the most complex and will likely take more time than the other steps. The above snippet should be a good starting point though.

Unfortunately, even after getting the web server configured, I had another problem. The default Gitweb web page is a bit… utilitarian, to phrase it mildly.

Gitweb uses a Perl CGI script to generate the HTML, which means unless you want to manually modify that script your basically stuck with the HTML that Gitweb gives you.

Fortunately, CSS is quite powerful these days and there is a lot you can do with it. With only 6.25 KB of CSS I was able to transform the drab, default Gitweb interface into what you see today at git.gpanders.com¹. You may think it still looks drab, but I’m fairly proud of how it turned out.

Gitolite tips and tricks

Here are some of the Gitolite tricks I’ve set up on my server that you may find useful.

Mirroring to GitHub

I still mirror most of my repositories to GitHub, as that’s still the best place for discoverability and compatibility with other programs.

This is easy to set up in Gitolite. In your gitolite.conf file, add the following to configure the repos you want to mirror to GitHub:

@github = repo1 repo2

repo @github
    config remote.github.url = git@github.com:gpanders/%GL_REPO.git
    config remote.github.mirror = true
    option hook.post-receive.mirror = mirror

Update your gitolite.rc file to allow the remote.* config options to be set:

    GIT_CONFIG_KEYS             =>  'remote\..*\.(url|mirror)',

Now create the following script at hooks/repo-specific/mirror in your local code path:

#!/bin/sh
git push --quiet --mirror --force github

You’ll also need to create an SSH key for your git user on your server and upload the public key to GitHub to allow your server to push to GitHub:

$ ssh yourserver.com
$ su - git
$ ssh-keygen -t rsa -b 2048 -C 'Git user'
$ # Paste the contents of ~/.ssh/id_rsa.pub to https://github.com/settings/ssh/new

Clone repos without .git suffix

GitHub and other popular hosting sites allow you to interact with remote git repos with or without the .git suffix, e.g.

git clone https://github.com/gpanders/dotfiles.git

and

git clone https://github.com/gpanders/dotfiles

are the same. Gitolite always uses the .git suffix on repositories, but you can simply create non-suffixed symlinks that will allow you to clone repositories with or without the .git extension:

$ cd /var/lib/git/repositories
$ ln -s dotfiles.git dotfiles

You can do this manually quite easily, but you can also create a Gitolite trigger to automatically create these symlinks everytime a new repo is created.

Create a trigger at triggers/post-compile/create-symlinks in your local code path with the following contents:

#!/bin/sh

# For projects that gitweb has access to, create symlinks in the 'repositories'
# directory without the .git suffix pointing to the real repos

symlink() {
    repo="$1"
    [ -L "$repo" ] && rm "$repo"
    if gitolite access -q "$repo" gitweb R any || gitolite git-config -q -r "$repo" gitweb\\.; then
        ln -s "$repo".git "$repo"
    fi
}

cd "$GL_REPO_BASE" || exit 1
if [ "$1" = "POST_CREATE" ] && [ -n "$2" ]; then
    # just one to be done
    symlink "$2"
else
    # all of them
    gitolite list-phy-repos | while IFS= read -r repo; do
        symlink "$repo"
    done
fi

In the above script, only repositories which are publicly acessible on Gitweb have a symlink created, since I only really care about the .git extension on the web interface (e.g. I want to use git.gpanders.com/dotfiles instead of git.gpanders.com/dotfiles.git). If you want this to apply to all repos, simply remove the gitolite access check.

You’ll also need a second trigger at triggers/post-compile/update-gitweb-access-list. This trigger is supplied by Gitolite by default, so creating a new trigger just overrides the default one. The only change we need to make to the default trigger is to exclude the .git suffix from the list of repositories that Gitolite makes available to Gitweb.

Copy the default trigger to triggers/post-compile/update-gitweb-access-list in your local code path and modify line 28 to

        echo "$repo" >> $tmpfile

Generate a README

GitHub and other sites automatically convert your README file (if it exists) from Markdown, reStructured Text, etc. into HTML when you visit the web page. We can do this too with Gitolite!

If the file README.html exists in the root of your repository, Gitweb will display it on the project’s summary page (example). We can create a hook to automatically create this file from the plain text README file in the repo. Create a hook at hooks/repo-specific/readme in your local code directory with the following contents:

#!/bin/sh

if ! command -v pandoc >/dev/null 2>/dev/null; then
        exit 0
fi

tmpdir=$(mktemp -d)
git --work-tree="$tmpdir" checkout --force

for file in README README.md README.rst README.txt; do
        if [ -f "$tmpdir"/"$file" ]; then
                pandoc -o README.html "$tmpdir"/"$file"
                break
        fi
done

rm -rf "$tmpdir"

Now in your gitolite.conf file add

repo @all
    option hook.post-receive.readme = readme

Obviously this requires that pandoc be installed on your git server.

Publish a Hugo site

This is the hook I use to publish this blog. I use Hugo to “compile” my site and I push it to GitHub where it’s hosted on GitHub Pages. My publish hook is located at hooks/repo-specific/publish:

#!/bin/sh

set -e

if ! command -v hugo >/dev/null 2>&1; then
        echo "hugo not found" >&2
        exit 1
fi

if [ -z "$GIT_DIR" ]; then
        echo "GIT_DIR not set" >&2
        exit 1
fi

TMPDIR=$(mktemp -d)
git --work-tree="$TMPDIR" checkout --force --recurse-submodules
git clone --quiet git@github.com:gpanders/gpanders.github.io "$TMPDIR"/public

echo "Building site..."
(cd "$TMPDIR" && hugo --quiet --cleanDestinationDir --destination public)

(
        cd "$TMPDIR"/public
        export GIT_DIR=.git
        if ! git diff-index --quiet HEAD --; then
                echo "Publishing site..."
                git add .
                git commit --quiet -m "Rebuilding site $(date)"
                git push --quiet origin master
                echo "Done!"
        fi
)

rm -rf "$TMPDIR"

and in my gitolite.conf I simply use:

repo blog
    option hook.post-receive.publish = publish

This is an archived link so you can see the CSS changes I’m referring to. The live version of https://git.gpanders.com no longer looks like this. ↩︎

Caching LDAP Queries in Mutt

Wed, 11 Mar 2020 00:00:00 +0000

You can use the lbdb tool to query an LDAP server within Mutt using the query_command setting. Setting up lbdb is out of scope for this article, although you can find plenty of documentation and examples online (here and here, for instance). One drawback to this approach, however, is that everytime you want to look up a recipient in Mutt you have to query your LDAP server, which can be quite slow.

A workaround to this problem is to “cache” the results of prior LDAP queries into a Mutt aliases file. That way you only have to query lbdb once, and future recipients can be found in the aliases file, which is significantly faster.

The following script does exactly this:

#!/bin/bash
# Query lbdbq and save results to Mutt alias file

set -e

# Mutt aliases file to save results to
ALIASES="$HOME/.cache/mutt/aliases"

# Only save email addresses from $DOMAIN. Leave empty to save all email addresses
DOMAIN="example.com"

query_and_cache() {
    results=$(lbdbq "$@" 2>/dev/null)

    printf '%s\n' "$results"

    # Remove first line from results
    results=$(sed '1d' <<< "$results")

    # Format results like mutt aliases
    sed -E $'s/^([[:alnum:]_.]+)@([[:alnum:]_.]+)\t([[:alnum:] -]+)/alias \\1 \\1@\\2 (\\3)/' <<< "$results" | awk -v domain="$DOMAIN" '$3 ~ domain {$2=tolower($2);print}' >> "$ALIASES"

    # Sort and remove duplicates
    sort -u -o "$ALIASES" "$ALIASES"

}
query_and_cache "$@"

The script queries your LDAP server using lbdbq and returns those “raw” results to Mutt so that you can make your selection. However, it also reformats the output and saves it to an aliases file for Mutt to use later.

Save this script to ~/.lbdb/query.sh, make it executable, and use it in Mutt’s query_command setting:

set query_command = "~/.lbdb/query.sh '%s'"

Now, everytime you make a query in Mutt the results will be saved to $HOME/.cache/mutt/aliases. Obviously, you need to tell Mutt to read from this alias file as well:

set alias_file = ~/.cache/mutt/aliases
source $alias_file

One caveat to this is that Mutt only reads the alias file on startup, so any changes to that file will not be detected unless you manually run :source $alias_file or restart Mutt.

Git Plumbing

Mon, 02 Mar 2020 00:00:00 +0000

Porcelain and Plumbing

git commit

Plain English: Create a new tree object from the current index and create a new commit object from that tree. The commit object combines a commit message, parent, date, author, committer, and the tree. Update the ref pointed to by HEAD with the hash of the new commit object.

Plumbing

Once files are in the index (using git add or git update-index) use git write-tree to create a new tree object:

$ git write-tree
8c3c7fbcd903744b20fd7567a1fcefa99133b5bc

Before creating the commit object, a commit message must be created. Usually, this is done by invoking the default editor. The file the editor opens is called COMMIT_EDITMSG in the .git directory and is pre-populated with the output of git diff-index --patch HEAD:

$ echo "\n# ------------------------ >8 ------------------------" > .git/COMMIT_EDITMSG
$ git diff-index --patch HEAD >> .git/COMMIT_EDITMSG
$ ${GIT_EDITOR:-${EDITOR:-vi}} .git/COMMIT_EDITMSG

The first line above is called a “scissor line” (the >8 looks like a pair of scissors). That line and everything below it is automatically deleted by git commit, but when using plumbing commands we must do it manually:

$ tmp=$(mktemp)
$ sed -n '/>8/q;p' .git/COMMIT_EDITMSG > "$tmp"
$ mv "$tmp" .git/COMMIT_EDITMSG

Finally, we can create the commit object.

$ tree="$(git write-tree)"
$ commit="$(git commit-tree -p HEAD -F .git/COMMIT_EDITMSG "$tree")"
$ git update-ref HEAD "$commit"

You Don't Want a Tax Refund

Thu, 13 Feb 2020 00:00:00 +0000

Tax time is approaching and this year, like every year, it is common to hear people excited about receiving their tax return discussing the exciting purchases they plan on making with this perceived windfall.

I want to challenge this conventional wisdom by stating a true but unpopular fact: getting a tax return is not necessarily a good thing and, in most cases, is actually the least desirable financial outcome one can achieve.

This non-intuitive fact can be understood by a quick change of perspective. Receiving a tax refund is equivalent to receiving repayment on a one year, interest-free loan to the government. Conversely, owing money during tax season is equivalent to paying back a one year interest-free loan to the government. When put in these terms, it seems obvious. So why don’t more people feel this intuitively?

The answer comes down to simple human psychology. Our fallible brains view the sudden influx of money that accompanies a tax refund as a windfall, similar to if we had just received a check from a distant relative or found a $20 bill on the ground. Additionally, paying taxes at the end of the year can feel like you’re losing money, and humans hate to lose things.

Many people “categorize” money into different bins: the money they earn from work as part of their paycheck goes towards regular, boring adult stuff like paying the rent and bills, while windfalls and gifts are tucked away for fun purchases, such as new clothes or a vacation. This is irrational and is generally considered bad financial practice, but that doesn’t stop us from doing it. The consequence of this is that, compounded with the loss of a year’s worth of interest, most tax refunds are spent on things that wouldn’t have purchased had that money been received as part of a normal cash flow. This ends up leaving people worse off, financially speaking.

Crunching the numbers

When it comes to tax season, the optimal outcome is actually to owe money, and the more the better! This is, again, counter-intuitive: who wants to owe money to the government? But remember that owing money during tax season means that you earned more than you would have otherwise during the rest of the year. Assuming you maintain good financial habits and are living below your means, that extra money was being deposited into an interest-bearing savings or brokerage account. When the tax bill comes due, you simply withdraw those funds to pay back the tax man. The kicker is that you get to keep all of the interest.

To put this into concrete numbers, let’s evaluate 3 different scenarios: 1) you owe nothing and are owed nothing at tax time (that is, you paid exactly the correct amount of taxes during the previous tax year); 2) you owe $3,000 at tax season; and 3) you receive a $3,000 tax refund.

(Again, for this analysis we are assuming that your income is in excess of your living expenses and that you’re making regular deposits into a savings account. If this doesn’t describe your situation then the following analysis won’t apply, although the conclusion remains the same.)

A $3,000 tax debit or credit means that each month you pocketed $250 more or less than you should have. To keep the math simple, we’ll assume that you normally save $250 a month from your income. For example, if you earn $5,000 a month, you spend $4,750 a month on living expenses and put $250 in the bank. So if you pay an extra $250 in taxes each month, you’re still spending $4,750 a month but saving nothing. If you pay $250 less in taxes each month, then you’re now saving $500 a month.

The conclusion that we’ll arrive at shortly doesn’t depend on the numbers I gave above: these are provided simply to make the argument a bit less abstract. Regardless of how much or how little you currently save or earn, the important point is that if you pay less in taxes you save more, and if you pay more in taxes you save less.

Let’s also assume that you’re a smart saver and you use a bank with a high-yield savings account (current annual yields for savings accounts as of this writing are around 1.8% on the high end). Compounded monthly, your balance earns 0.15% interest each month.

When saving $250 a month, at the end of one year your bank account will have $3,024.87. Saving $500 a month gets you $6,049.75.

At the end of the year you file your taxes. In the first scenario, you break even and owe nothing and also receive no refund. You walk away with $3,024.87. In the second scenario, we owe $3,000 in taxes, which we simply subtract from our savings account, leaving us with $3,049.75. And finally in our third scenario, we have nothing in our savings account but we get a $3,000 tax refund, so we of course walk away with $3,000.

All together:

Break Even	Taxes Owed	Tax Refund
$3,024.87	$3,049.75	$3,000.00

Now you might say: “Okay, so there’s a difference of less than $50. Big deal. Over one full year that’s not much”. Let’s see what happens if, instead of a high-yield savings account, we invested in a US stock index fund (which historically averages about 7%):

Break Even	Taxes Owed	Tax Refund
$3,098.15	$3,196.29	$3,000.00

As you can see, the difference is much more dramatic.

The amount of taxes owed or refunded also matters. In our example, we assumed a $3,000 debit or credit, but if that had been say, $10,000 the difference would have been even more dramatic, providing $165.83 in a high-yield savings account and over $650 in a stock index fund.¹

To put this in finance terms, this is essentially buying on margin using funds borrowed from the government. Except, unlike in the finance world, this borrowing incurs no interest and there is never a margin call – it is essentially risk-free. If you have the financial stability to do so, this is not a bad way to (hyper) optimize your finances to earn a little extra money each year.

So this tax season, cross your fingers and hope that you get to write a check instead of receiving one, and smile to yourself knowing that math is on your side.

It’s important to remember that stock index funds don’t always provide a 7% return; that is simply the historical average. In 2019, a stock index fund yielded a return of 30.8% which would have resulted in an astounding $923.83 of interest for scenario 2; however, in a down year this is a very easy way to lose money. ↩︎

Using Xilinx Vitis for Embedded Hardware Acceleration

Wed, 12 Feb 2020 00:00:00 +0000

Xilinx recently released their new Vitis tool, which aims to ease the process of accelerating high-level algorithms in applications in an FPGA. It is an ambitious tool with a lot of potential. This guide will help you get started.

The guide is targeted toward the Zynq Ultrascale+ MPSoC using a command line (as opposed to a GUI) flow because that is what I use. However, where possible, I’ve aimed to keep things as device agnostic as possible.

NOTE: Vitis is still a very new tool and is likely to change rapidly in the near-future. I will try to keep this guide as up-to-date as possible, but be warned that some pieces may be antiquated by the time you read it.

You can find a “reference implementation” of the steps below here. This implementation uses a Makefile to automate all of the steps outlined below with a simple example design. You are welcome to copy the reference implementation and modify it to your own needs however you wish.

You can also find a lot of examples and Vitis tutorials online provided by Xilinx. However, almost all of these are targeted towards using x86/PCIe platforms and do not carry over well into edge-based/Zynq platforms (hence the need for this guide).

Outline

The high-level outline of doing hardware acceleration in Vitis is

Pre-packaged Embedded Platforms

Xilinx offers a set of standard embedded hardware platforms available here. If you’re just getting started or if your design does not contain any custom IP or infrastructure, you can use one of these standard platforms and skip steps 1-3 above.

To use one of these platforms (for example, zcu102_base), download the ZIP file from Xilinx’s website and extract the platform to the platforms subdirectory beneath your Vitis installation. For example, if you installed Vitis to /opt/Xilinx/Vitis/2019.2, you would copy the zcu102_base directory to /opt/Xilinx/Vitis/2019.2/platforms/zcu102_base. In the rest of this guide, you will then provide zcu102_base as the argument to the --platform command line flag.

Also download the cross-compilation sysroot from the same downloads page. After extracting the archive you’ll find sdk.sh scripts for both the Zynq 7000 and the Zynq Ultrascale. Execute the sdk.sh script for your chip and supply an installation path for the sysroot.

UPDATE 2020-06-23: The 2020.1 Vitis update seems to have removed the sections on creating custom platforms from their documentation (at least, I haven’t been able to find it). I am not sure what the reason is behind this. Perhaps they want users to stick to their pre-packaged embedded platforms? In any case, the instructions below still seem to be valid, but proceed with caution.

Creating Your Hardware Design

NOTE: You can skip this step if you are using a pre-packaged embedded platform.

This step is done using Vivado and is responsible for generating the Xilinx Shell Archive (xsa) file (formerly known as a Hardware Description File (hdf)). Your hardware design needs to include the Zynq processor IP as well as at least one external clock. You can find a simple example in Xilinx’s documentation.

Each clock also must have an accompanying Processor System Reset IP and a PFM.CLOCK property that can be set either in the Vivado GUI (click Window > Platform Interfaces) or in the Tcl console:

set_property PFM.CLOCK { \
    <clk port> { \
        id "0" \
        is_default "true" \
        proc_sys_reset <proc_sys_reset name> \
        status "fixed" \
    } \
}

Every platform must specify one clock with id=0, status="fixed" and is_default="true".

In addition to the clocks, you must also specify the available memory ports in your design. Again, this can be done in the GUI in the Window > Platform Interfaces window or can be done directly in Tcl:

set_property PFM.AXI_PORT { \
    <port_name> {memport <type> sptag <ID> memory <value>} \
}

The sptag and memory parameters are optional. For a full description of these properties, see the Configuring Platform Interface Properties page.

The platform interfaces defined in this stage determine how Vitis will connect the memory interfaces of your kernels.

Create Linux Software Components

NOTE: You can skip this step if you are using a pre-packaged embedded platform.

Vitis requires the following software components:

First Stage Bootloader (fsbl.elf)
PMU Firmware (pmufw.elf)
U-Boot (u-boot.elf)
ARM Trusted Firmware (bl31.elf)
Linux kernel image, device tree blob, and initramfs (image.ub)

Note that it is not required that your Linux kernel be packaged with the device tree blob and initramfs into an image.ub file, but that is what the tools are set up to use by default. The image.ub file is a FIT image file that combines the Linux kernel image, the device tree blob, and a root file system together into a single file.

The easiest way to generate all of these components in a way that will work basically out of the box with Vitis is to use Xilinx’s PetaLinux tool. Note that it is NOT required to use PetaLinux, and there are many very good reasons not to do so, but again for the sake of brevity and clarity this guide will assume the use of PetaLinux.

If you choose to use PetaLinux, you can follow the instructions here.

The most important things to notice about the instructions listed above are the inclusion of userspace packages in the rootfs (xrt, zocl, opencl-clhpp, and opencl-headers) and the modification of the device tree. Namely, you must have the following somewhere in your device tree source file:

&amba {
    zyxclmm_drm {
        compatible = “xlnx,zocl”;
        status = “okay”
    };
};

Without this addition, the zocl driver will not be loaded and the Xilinx Runtime will not be able to detect your hardware device.

If you use plain YoctoLinux, the xrt and zocl applications can be found in Xilinx’s meta-petalinux layer.

One other important modification you must make is to disable the CONFIG_CPU_IDLE kernel option. See AR# 69143 for more information. Without this modification, QEMU will hang during bootup (UPDATE 2020-04-14: This step is now included in the official Vitis documentation. See Step 9 here).

Once you run petalinux-build, you will find all of the requisite software components in the images/linux/ directory. Copy these to a location of your choice (e.g. a boot subdirectory within your project directory). You will also need to extract the rootfs.tar.gz archive file. This file contains the sysroot that will be installed onto your target. For example, if our project directory is located at ~/Projects/vitis_example/:

mkdir -p ~/Projects/vitis_example/build/{boot,sysroot}
cp images/linux/{image.ub,zynqmp_fsbl.elf,pmufw.elf,u-boot.elf,bl31.elf} ~/Projects/vitis_example/build/boot
tar -C ~/Projects/vitis_example/build/sysroot -xf images/linux/rootfs.tar.gz

You will also need to include a BIF file, which is a file which tells Xilinx’s bootgen tool how to generate the BOOT.bin file that is used by MPSoC’s boot ROM to boot the device. The file should have the following contents:

/* linux */
the_ROM_image:
{
       [fsbl_config] a53_x64
       [bootloader] <zynqmp_fsbl.elf>
       [pmufw_image] <pmufw.elf>
       [destination_device=pl] <bitstream>
       [destination_cpu=a53-0, exception_level=el-3, trustzone] <bl31.elf>
       [destination_cpu=a53-0, exception_level=el-2] <u-boot.elf>
}

The file names within the <> brackets will be expanded automatically by Vitis, so there is no need to insert absolute paths in this file. Save the BIF file as linux.bif in your boot directory.

Finally, you will need two plain text files that provide the command line arguments to QEMU. You can simply copy these from Xilinx’s [Vitis GitHub page] and save them to your boot directory. Note that, unfortunately, the names of these two files do matter: they should be named qemu_args.txt and pmu_args.txt respectively.

Vitis uses these software components to run the software and hardware emulation targets, which we’ll get to later.

Generate a Xilinx Platform File

NOTE: You can skip this step if you are using a pre-packaged embedded platform.

Vitis introduces some new jargon: platforms, domains, and system projects. A platform is essentially the hardware platform which we created in step 1. Each platform has one or more domains. A domain is the BSP or OS that controls a group of processors in the hardware. A system project is a container for multiple applications that run on different domains at the same time.

In our example, the domain is simply Linux running on the ARM Cortex A53 processor. You can create the platform file in the Vitis GUI by following the instructions here or you can simply run the following commands from xsct (assuming you’re currently in your project directory):

platform create -name vitis_example -hw /path/to/vitis_example.xsa -proc psu_cortexa53 -os linux -no-boot-bsp -prebuilt -out ./build/platform
domain config -image ./build/boot
domain config -sysroot ./build/sysroot
domain config -boot ./build/boot
domain config -bif ./build/boot/linux.bif
domain config -qemu-args ./build/boot/qemu_args.txt
domain config -pmuqemu-args ./build/boot/pmu_args.txt
domain config -qemu-data ./build/boot
platform generate

This will create an xpfm file in build/platform/vitis_example/export/vitis_example/ alongside two directories: hw and sw. If you copy or move the xpfm file, you must also move the hw and sw directories, as the xpfm file depends on these two directories and expects them to be adjacent to itself.

Write and Compile Your Kernels

Writing OpenCL or Vivado HLS kernels is a huge topic that is beyond the scope of this guide. As a simple example, however, assume we have the following multiply-and-add kernel at kernels/axpy/axpy.c:

void axpy(float const *a, float const *x, float const *y, float *out, int const len)
{
#pragma HLS INTERFACE m_axi port=a offset=slave
#pragma HLS INTERFACE m_axi port=x offset=slave
#pragma HLS INTERFACE m_axi port=y offset=slave
#pragma HLS INTERFACE m_axi port=out offset=slave
#pragma HLS INTERFACE s_axilite port=a bundle=control
#pragma HLS INTERFACE s_axilite port=x bundle=control
#pragma HLS INTERFACE s_axilite port=y bundle=control
#pragma HLS INTERFACE s_axilite port=out bundle=control
#pragma HLS INTERFACE s_axilite port=len bundle=control
#pragma HLS INTERFACE s_axilite port=return bundle=control

    for (int i = 0; i < len; i++) {
#pragma HLS PIPELINE
        out[i] = a[i]*x[i] + y[i];
    }
}

The first step is to compile our kernel into a Xilinx object file (.xo):

mkdir -p build/sw_emu
v++ --platform ./build/platform/vitis_example/export/vitis_example/vitis_example.xpfm -t sw_emu -g -o build/sw_emu/axpy.xo -c kernels/axpy/axpy.c

Note that the xpfm file created in the last step is a required argument to the v++ compiler.

If you are using a pre-packaged embedded platform, supply the name of the platform as the argument to the --platform command line flag. For example,

v++ --platform zcu102_base ...

Once you have one or more .xo files, you can link them together into an .xclbin file:

v++ --platform ./build/platform/vitis_example/export/vitis_example/vitis_example.xpfm -t sw_emu -g -o build/sw_emu/axpy.xclbin -l build/sw_emu/axpy.xo

Also note that we passed the -t sw_emu option to v++ in both the compile and link phases. The -t option is mandatory and determines what actually is produced in the .xclbin file. The available options are sw_emu, hw_emu, or hw. For now, we’ll just use sw_emu (meaning “software emulation”).

We now have our platform file and our xclbin file. All that’s left is to write and compile the host code and test our application in the emulator.

For more information on using v++ see the official Xilinx documentation.

Write and Compile the Host Code

Again, this step is out of scope for this guide as it is highly design dependent. The easiest way to get started on this step is to start from an example.

Note that as of this writing (Feb 2020) Xilinx only supports OpenCL 1.2. This is in part because Xilinx depends on some APIs that were deprecated in OpenCL 2.0. You can find the OpenCL 1.2 reference pages here.

Run Software Emulation

This is the point where the edge-based flow differs significantly from an x86/PCIe platform. In order to do software emulation for the ARM CPU, Vitis spins up a QEMU virtual machine using the parameters supplied during platform creation. At this point, you can run your host executable with the compiled xclbin file. The Xilinx Runtime will generate run summaries and reports on the target VM, which you must then transfer back over to your host development machine.

The software emulation VM is launched using a script called launch_emulator. When you source the Vitis settings64.sh file, this script is added to your path. When you run launch_emulator, the script looks for files under the _vimage directory, which is created during the v++ linking phase. This directory contains parameters used by the launch_emulator script to prepare and start the QEMU VM.

The first thing this script does is prepare a virtual SD card image which is passed to QEMU. A file called sd_card.manifest tells the launch_emulator script what files should go on this SD card image. Unfortunately, by default this manifest file does not include all of files needed to run software emulation. Before running launch_emulator, you will need to modify the sd_card.manifest file to include the absolute path to your host executable as well as any other files you want to include in the QEMU VM.

You should also include a xrt.ini file with the following contents:

[Debug]
profile=true
timeline_trace=true
data_transfer_trace=fine

This will generate useful output products when you run the emulation. Be sure to include the full path to this xrt.ini file in the sd_card.manifest file.

Once the sd_card.manifest file is ready, run the following command to launch the emulator:

launch_emulator -no-reboot -runtime ocl -t sw_emu -forward-port 1440 1534

The -no-reboot parameter is passed to QEMU and means that instead of rebooting, the VM will simply shutdown. The -runtime and -t flags are used by the launch_emulator script itself. The -forward-port flag creates a port forward to the guest VM allowing you to connect to it using xsct.

If everything works correctly, you should see the VM booting up in your terminal console. Eventually, you will reach a login prompt. The username and password are both root. Once logged in, you can mount the virtual SD card and run your host executable:

mount /dev/mmcblk0p1 /mnt
cd /mnt
export XILINX_XRT=/usr
XCL_EMULATION_MODE=sw_emu ./host vitis_example.xclbin

You can use xsct to transfer files between your development machine and the guest VM:

$ xsct
xsct% connect -url tcp:localhost:1440
xsct% tfile copy -from-host /path/on/host /path/on/target
xsct% tfile copy -to-host /path/on/target /path/on/host

This allows you to make changes to your host program or xclbin file and quickly transfer them to the VM without needing to restart the emulator. This is also how you can transfer the run summaries off of the target VM onto your host:

$ xsct
xsct% connect -url tcp:localhost:1440
xsct% tfile copy -to-host /mnt/profile_summary.csv profile_summary.csv
xsct% tfile copy -to-host /mnt/timeline_trace.csv timeline_trace.csv
xsct% tfile copy -to-host /mnt/xclbin.run_summary xclbin.run_summary

The xclbin.run_summary file can be viewed using the vitis_analyzer tool:

$ vitis_analyzer xclbin.run_summary

Conclusion

There you have it. There are a lot of steps involved, but fortunately almost all of them are entirely scriptable (as you can see in the reference implementation). This means that once the process is done once, the time cost of repeating it is negligible.

If you have any questions or feedback, please feel free to contact me.

Archive the Internet

Fri, 07 Feb 2020 00:00:00 +0000

As the Internet continues to grow and age, we are seeing more and more of a phenomenon called “link rot”. Link rot is the all-too-familiar experience of returning to a bookmark or link only to find that it’s no longer maintained, been taken down, or even changed to something else. What can we do about this?

Sites like archive.org were created to combat this exact problem. That site exists for the sole purpose of maintaining snapshots of the Internet over time so that you can always find old copies of a web page from years back.

The problem, of course, is that you never know if the site you actually need will be archived because they only archive sites that are explicitly submitted by users. For popular sites, this won’t be a problem. But what about an obscure blog post or research article you read 4 years ago that you now want to reference? If you didn’t remember to archive it at the time, you’re out of luck.

Other sites like Pinboard offer an automatic archiving feature for all of your bookmarks. Pinboard is a great, well-run, and honest service that I use and recommend. For a fee, Pinboard will regularly download a copy of your bookmarks to save them from link rot.

The solution to this problem is actually quite easy, technically speaking. I’ve been itching for a new project, so I thought this would be a good problem to tackle.

The system I wanted to create has basically 3 components: input, retrieval, and presentation. The input component relates to how I specify which web pages I want to archive. Retrieval involves actually downloading those pages in such a manner that they can be accurately duplicated. Presentation deals with viewing the archive when needed.

All three of these components need to work with a common data source. In the interest of simplicity and to avoid reinventing the wheel, the obvious choice is to use some kind of plain-text format to represent which web pages I want to archive. This ensures that I can leverage standard UNIX command line utilities such as wget and awk.

I first considered GNU’s recutils, a plain-text database format that supports inserting and selecting records into “recfiles”. The format is brain dead simple and easy to understand, and since I’m not going to be carrying around millions of entries (like SQL databases are made for), the performance factor is not really a concern for me.

The only drawback to recutils is support, or lack thereof. The presentation component of this system will need to involve a webserver of some kind and recfiles are not a natively supported database format in any language I am aware of. Integrating recfile support into a webserver would involve either parsing the files using regular expressions (hacky and non-portable) or writing a recfile driver myself (a lot of work and I’m lazy). Instead, I decided to use a tab-delimited CSV file.

My tab-delimited CSV database has a very simple format: it’s a single file where each line in the file has two fields separated by a tab character. The first field is the web page title or description, and the second field is the URL. For example

Personal website of Gregory Anders	https://gpanders.com
                                  ^ literal tab character (\t)

Why a tab? Traditionally, CSV files use commas to separate field values (hence the C in CSV); however, page titles and descriptions can contain basically any printable character imaginable, including commas, semicolons, pipes, and slashes. But they don’t contain raw tab characters, so there will never be any ambiguity. This format is easy to read, easy to manually update (if I want), works perfectly with traditional UNIX tools, and is widely supported (many languages come with native CSV support with no 3rd party dependencies).

To populate the database, I import my bookmarks from Pinboard and convert them into the tab-delimited CSV format:

$ curl -s "https://api.pinboard.in/v1/posts/all?auth_token=$PINBOARD_API_KEY&format=json" | jq -r '.[] | "\(.description)\t\(.href)"' > links.csv

Then, I use awk to parse the URLs from the CSV file and wget to download the web pages:

$ awk -F'\t' '{print $2}' links.csv | while read -r url; do
>   wget -ENHkpc --wait=1 --random-wait --user-agent="" -e robots=off "$url"
> done

To build the wget command I referred to a few different examples across the web. The flags given will download all of the resources on the given page and adjust their paths accordingly so that the web page can be viewed offline. The --wait and --random-wait flags tell wget to wait between 0.5 and 1.5 seconds between each download request which reduces the load on the servers a bit.

For the presentation component, I wrote an ultra-simple Go web server that simply serves up the web archive and provides a dynamic index of archived pages from the database. The entire program is fewer than 100 lines and you can find it here.

By setting up a cron job on my Raspberry Pi, I can now completely automate the creation and maintenance of an offline web archive (using Borg for regular backups!). Bookmarks are automatically imported from Pinboard into a CSV file which is read by standard UNIX tools awk and wget to download a web archive, which is viewable at any time from a webserver.

This solution is extremely stable as it uses technologies and tools that have been around for decades. By leveraging the UNIX philosophy of using simple, composable tools with flexible interfaces, the system is highly modular and configurable (I can easily modify or replace the web interface without having to do anything to the other components, and vice versa). And since I’m backing up the whole archive to Borg, I can maintain a weekly snapshot of all of my bookmarks in perpetuity at the cost of very little storage space.

This whole thing took me a little over 6 hours to create. Most of that time was spent deciding on a database format and building the Go webserver (like I said, it’s extremely simple but I am also a total Go newbie, so it was a great learning experience). Future improvements involve tweaking up the web interface to make it a bit more aesthetically appealing and adding some functional improvements such as a search feature.

2019 Reading Review

Sat, 28 Dec 2019 00:00:00 +0000

In 2019, I read 27 books. This was less than last year’s 40 and is not an overly impressive quantity for any serious reader, yet I’m happy with both the amount and the content that I read this year. Below, I’ll give a brief summary of my favorites.

In January, I read Daniel Kahneman’s Thinking, Fast and Slow. This is an extraordinary book written by one of the world’s most famous psychologists that discusses in depth and in riveting detail the numerous cognitive and behavioral biases that human beings exhibit. Kahneman and his life-long academic partner Amos Tversky revolutionized the world of behavioral psychology and behavioral economics by demonstrating empirically that humans behave and make decisions in systematically biased ways. This book is a must-read for decision makers and anyone interested in understanding their own psychological flaws and blind spots.

After Thinking, Fast and Slow, I read Michael Lewis’ The Undoing Project, a biography of both Kahneman and Tversky that covers much of the material in Thinking, Fast and Slow. I enjoy Lewis’ writing and have read almost everything he has written. The Undoing Project offers a more “lightweight” introduction to the concepts in Thinking, Fast and Slow along with Lewis' excellent narrative spin.

Gates of Fire by Steven Pressfield is a historical fiction novel recounting the famous Battle of Thermopylae (the same battle portrayed in the glaringly inaccurate 300). Pressfield’s novel is told through the eyes of one of the free Greeks living in Sparta who joins the 301 Spartiates (King Leonidas plus 300 peers) to stymie the advance of Xerxes’ Persian Army at the narrow pass of Thermopylae (Greek for “The Hot Gates”). Pressfield’s telling is gripping and tragic. He paints a bleak picture of the violence and chaos that are the hallmarks of all warfare, but particularly the up-close clashes of ancient battles. Pressfield’s novel does not overly valorize Spartan culture, but it certainly does omit many of the more unseemly aspects. For more on this, I highly recommend Bret Devereaux’s series on Spartan culture.

Tom Holland’s Rubicon was one of my favorite books of 2019. Holland recounts the twilight years of the Roman Republic, from around 120 BC to 20 BC. Holland’s narrative storytelling is utterly compelling and one could be forgiven for forgetting that they are reading actual history. Rome’s history is packed with intrigue, suspense, plots, twists and turns, and outrageous characters, and Holland does a remarkable job of bringing this world to life. It is incredible to think that some of the world’s most famous historical characters, such as Julius Caesar, Pompey, Cicero, Cato, Octavian, Mark Antony, Cleopatra, and others, all lived at the same time. Rubicon tells the story of one of the most dramatic periods in human history and I highly recommend it.

Two books I read on recommendations from the same friend were Einstein’s Dreams by Alan Lightman and The Invention of Nature by Andrea Wulf. Einstein’s Dreams is a short collection of fictional vignettes representing Einstein’s own visions of time during the period he was developing his theory of special relativity. Lightman is both a physicist and a writer. His familiarity with the underlying scientific concepts is beautifully blended with his prose to create a unique and thoroughly enjoyable reading experience.

The Invention of Nature is a biography of Alexander von Humboldt, the most famous scientist whom I had never heard of. Humboldt was perhaps the most famous scientist in the world during his lifetime (1769-1859) and while he did not discover or invent any particular mathematical or scientific theory on which to attach his name, he is almost single-handedly responsible for defining the current scientific and popular conception of nature as a complex, interrelated web-of-life.

It is hard to believe, but prior to Humboldt the popular thinking in Enlightenment circles was that nature was a mechanistic process, almost like a clock. It was perfectly and rationally designed, with pieces that operated independently from each other. Descartes thought of animals as closer to machines or automata rather than creatures similar to humans. Humboldt, born in Prussia, traveled throughout South and Central America and through his writings, drawings, and samples brought a vivid picture of that world to the scientific community back in Europe. His observations laid the foundations for many scientists who followed: Charles Darwin read Humboldt obsessively and corresponded with him later in life. Humboldt also inspired other famous environmentalists and naturalists such as George Perkins Marsh (the founding father of modern environmentalism), Henry David Thoreau, and John Muir.

This year I also continued my Stoic education, reading Seneca’s Letters from a Stoic (also known as the Epistulae Moralis), Marcus Aurelius' Meditations, and Epictetus’ Enchiridion (I read the version titled How to be Free, which is essentially just a reprint of the Enchiridion). My study of Stoic philosophy has been the chief reason behind my ever-increasing sense of calm and self-control and has been the best tool I have found for battling feelings of anxiety and despair. These three books represent perhaps the “trinity” of Stoic thought, and I recommend them to anyone interested in learning more about Stoic philosophy (as a starting place, however, I highly recommend Will Irvine’s Guide to the Good Life).

On a recommendation from Brett McKay of The Art of Manliness, I picked up Lonesome Dove by Larry McMurtry this year and absolutely loved it. This is perhaps my favorite book that I read this year and is certainly now one of my favorite works of fiction. Lonesome Dove tells the tale of a cattle drive from South Texas to Montana set sometime in the 1870’s. McMurtry’s story telling ability is peerless and the characters he creates are funny, tragic, inspirational, frustrating, and terrifying. McMurtry’s knowledge of Western geography and history is evident throughout the tale and no other work of fiction captures the gritty and unromantic aspects of this hard life quite so well. I recommend this book to anyone, but particularly to those who grew up in or are enamored with the American West.

From Lonesome Dove, I picked up T.R. Fehrenbach’s Lone Star, his history of Texas and the Texans (this book was also recommended by Ryan Holiday, who’s book recommendations I follow religiously). Growing up in Texas, I took Texas history in grade school, which to me now seems utterly inappropriate. More than any other state, the settlement of Texas was steeped in violence and bloodshed. Texas was fought over fiercely by three sides, all of whom mistrusted and disliked each other: the Anglo-Saxon settlers from the United States, Spanish (and later Mexican) settlers, and the Plains Indian tribes, most notably the Comanches, Kiowas, and Apaches. Texas’ history is tragic, heroic, dramatic, and fascinating. The type of people who were willing to scrabble out a hard-won existence on Texas soil helped define the Texan ethos and culture which persists to this day. Lone Star gave me a somber appreciation and respect for Texas’ history.

S.C. Gwynne’s Empire of the Summer Moon, a fascinating telling of the incredible story of Cynthia Ann Parker and her son Quanah Parker, continued the theme of Texas history. Cynthia Ann was abducted by Comanche Indians while she was a child from her family’s farm, where most of her family were killed. She was raised as a Comanche and later gave birth to three children, one of whom was named Quanah Parker, who would go on to become one of the most famous American Indians in U.S. history. Empire of the Summer Moon tells the brutal and captivating history of the clash between Texan settlers and the Comanche Indians. Gwynne is not shy about confronting the uncomfortable fact of Comanche brutality and cruelty. Reality is complex, and this book paints the complicated picture of that era. The Comanche Indians are portrayed as a sympathetic group to the extent that we see their culture and lifestyle slowly eroded by the settlement of Texas; however, their own violence, cruelty, and brutality make them more than mere victims. Empire of the Summer Moon, like Lone Star, forces the reader to reckon with the complications of human nature and the violent terror of Texas’ past.

I enjoyed Empire of the Summer Moon so much that I read Gwynne’s newest book Hymns of the Republic, the narrative history of the final year of the American Civil War. Gwynne is an exquisite author, and his ability to construct a compelling story line from historical events is unmatched. Hymns of the Republic draws into focus just how truly tragic the American Civil War was and also presents the stark realization that history is not nearly as clean as we would often like it to be. Both the Union and the Confederacy had their share of incompetent, racist, and corrupt leaders, as well as their share of courageous and valorous men who thoroughly believed they were doing the right thing. At the beginning, the Union’s motivation for the war was emphatically not about abolishing slavery or black liberation, but merely about re-uniting the country (it is unquestionable, however, that the South’s primary concern beneath their overtures of “states’ rights” and their reason for seceding was the continuance of slavery). Abraham Lincoln himself stated that if he could reunite the Union without abolishing slavery, he would do it. However, his own personal disdain for the institution as well as ever-increasing pressure from Northern abolitionists (particularly Frederick Douglass) finally compelled Lincoln to pivot his stance, refusing to even consider reunification without abolition. He suffered dearly for this position. Lincoln was deeply unpopular and was one of the most hated men in America. In the North, he was hated either for doing too much (many Americans were not happy with the fact that their sons, fathers, and brothers were dying to free slaves, which they feared would unleash a wave of black migration into the North) or not doing enough. Hymns of the Republic recounts all of the political and military drama of that era and sheds some of the mysticism around the characters of that time. Like Holland’s Rubicon, Hymns of the Republic so effectively draws you in to the story that it often makes you forget that you are reading real history. S.C. Gwynne has entered the list of authors of which I will read anything they write.

I read Isaac Asimov’s Foundation in a single sitting. I consider myself a fan of science fiction and yet had never read this seminal series. Needless to say, it did not disappoint. I love Asimov’s writing style (the stylistic structure of Foundation has echoes of one of my favorite short stories, The Last Question) and was completely enraptured by the tale of the waning Galactic Empire and the nascent, underdog Foundation.

Endurance by Alfred Lansing is the true story of Ernest Shackleton’s doomed Imperial Trans-Antarctic Expedition. In 1914, Ernest Shackleton and his crew set out to become the first to cross the Antarctic continent overland. However, on their way to the continent their ship became trapped in the pack ice and eventually sank. In Endurance, Lansing tells the incredible tale of how Shackleton led his men safely back to civilization.

Finally, three of the most compelling books I read this year were Cal Newport’s So Good They Can’t Ignore You, Deep Work, and Digital Minimalism. The first contains a novel take on traditional career guidance, the premise essentially being that “follow your passion” is worthless and even harmful career advice. Many people don’t have “passions” that can readily be turned into a career, nor has doing something you’re passionate about been found to actually result in career satisfaction. Instead, the things that define a satisfying career are autonomy, connectedness, and competence. The logical conclusion is to instead focus on developing skills and experience (what Newport calls career capital) that you can use to leverage positions that offer these important traits.

Deep Work and Digital Minimalism go hand-in-hand and are Cal Newport’s manifestos on the importance of focus and attention in an increasingly distracted world. Newport, a computer science professor at Georgetown, is utterly practical in his writing style, eschewing theoretical pondering and instead offering concrete practices, methodologies, and advice supported by research and empirical data. But don’t be deceived - his writing style is entirely approachable and conversational and the ideas can be digested by anyone. Deep Work is, in my opinion, required reading for any knowledge worker who wishes to be successful in today’s economy.

Evaluating Markdown Code Blocks in Vim

Mon, 12 Aug 2019 00:00:00 +0000

When writing Markdown it is common to have code blocks within your document that look something like this:

This is a code block:
```bash
echo "Hello, world!"
```

One of the great features of Emacs’ Org mode is its ability to evaluate blocks of code from within the editing environment. I always appreciated this feature about Org mode which got me to thinking about implementing something similar in Vim.

Update (2020-03-02): I’ve packaged this into a Vim plugin called Medieval. The remainder of this post contains the original implementation, which is now out-of-date.

The solution ended up being fairly straightforward. I created the following autoloaded function in ~/.vim/autoload/ft/markdown.vim:

function! ft#markdown#eval()
  let view = winsaveview()
  let line = line('.')
  let start = search('^\s*[`~]\{3,}\S*\s*$', 'bnW')
  if !start
    return
  endif

  call cursor(start, 1)
  let [fence, lang] = matchlist(getline(start), '\([`~]\{3,}\)\(\S\+\)\?')[1:2]
  let end = search('^\s*' . fence . '\s*$', 'nW')
  let langidx = index(map(copy(g:markdown_interp_languages), 'split(v:val, "=")[0]'), lang)

  if end < line || langidx < 0
    call winrestview(view)
    return
  endif

  if g:markdown_interp_languages[langidx] !=# lang
    let lang = split(g:markdown_interp_languages[langidx], '=')[1]
  endif

  let block = getline(start + 1, end - 1)
  let tmp = tempname()
  call writefile(block, tmp)
  echo system(lang . ' ' . tmp)
  call winrestview(view)
endfunction

Basically, the function looks backward from the current cursor position for any line that matches the start of a code block (three or more back ticks followed by a language name) and then looks forward for the close of the code block. If the current cursor position is between those two locations, then we know we’re in a code block. At that point, it’s as simple as copying the text within the code block to a temp file and sourcing it using the interpreter given by the syntax of the code block (e.g. bash, ruby, etc.).

Obviously, not all languages can be run in an interpreter, so there is also a check to make sure the language of the code block is in a user-specified variable g:markdown_interp_languages. I set this in my Markdown ftplugin file (~/.vim/after/ftplugin/markdown.vim):

let g:markdown_interp_languages = ['python', 'sh=bash', 'bash', 'ruby', 'console=bash']

The = syntax lets me specify that certain syntaxes use the same interpreter (e.g. sh and console syntaxes should both use bash as their interpreter). Adding more languages is as simple as extending this global variable.

Finally, I map this function to a key binding in my ftplugin file:

nnoremap <silent> <buffer> Z! :<C-U>call ft#markdown#eval()<CR>

Now I can press Z! in a code block in Markdown and have the block evaluated.

This is a very lightweight solution and doesn’t have anywhere near the full features of Org mode (which includes things like inline results, persistent contexts, mangling, and more), but it’s a good first cut solution which I’m happy with.

As always, you can find the most up-to-date version of the function above, as well as all of my Vim files, here.

Write Your Own Man Pages

Mon, 29 Jul 2019 00:00:00 +0000

This post was inspired by the tool um. I decided to try my hand at making my own lightweight implementation using just a shell function. I decided to name my tool ual as a nod to the fact that it complements man.

I wanted to be able to write quick, simple notes in Markdown and be able to view them in a man-page style format from anywhere on the command line. Additionally, I wanted to be able to edit or create new notes with little to no “friction”.

Pandoc is able to convert Markdown files into Roff format (the format used by man pages), and the groff tool can then convert that into the actual formatted text that is displayed in the terminal. So converting a Markdown file into a man page is as simple as

$ pandoc --standalone --from markdown --to man my-note.md | groff -T utf8 -man | less -R

This is simple enough to turn into a shell function:

ual() {
   pandoc --standalone --from markdown --to man "$1".md | groff -T utf8 -man | less -R
}

To edit or create new notes, simply add an edit subcommand:

ual() {
    case $1 in
        edit)
            "${EDITOR:-vi}" $2.md
            ;;
        *)
            pandoc --standalone --from markdown --to man "$1".md | groff -T utf8 -man | less -R
            ;;
    esac
}

And of course, you probably want to keep all of those notes in a central location.

NOTES="$HOME/.notes"
ual() {
    case $1 in
        edit)
            ${EDITOR:-vi} "$NOTES"/$2.md
            ;;
        *)
            pandoc --standalone --from markdown --to man "$NOTES"/"$1".md | groff -T utf8 -man | less -R 
            ;;
    esac
}

Now I can use ual edit <note> to modify or create a new note, and ual <note> to view it.

I don’t want to include metadata in the actual Markdown notes themselves so that I can keep them as minimal (and portable) as possible. So instead I generate the metadata in the ual function itself.

NOTES="$HOME/.notes"
ual () {
    case $1 in
        edit)
            ${EDITOR:-vi} "$NOTES"/$2.md
            ;;
        *)
            NOTE="$NOTES"/$1.md

            if [ ! -f "$NOTE" ]; then
                echo "No ual entry for $1" >&2
                return 1
            fi

            TITLE="$(echo $1 | tr '[:lower:]' '[:upper:]')"
            SECTION="ual"
            AUTHOR="Gregory Anders"
            DATE="$(date +'%B %d, %Y' -r "$NOTE")"

            pandoc \
                --standalone \
                --from markdown \
                --to man \
                --metadata title="$TITLE" \
                --metadata author="$AUTHOR" \
                --metadata section="$SECTION" \
                --metadata date="$DATE" \
                "$NOTE" | groff -T utf8 -man | less -R
            ;;
    esac
}

As an example, if I have the following Markdown note at ~/.notes/foobar.md:

# NAME

foobar - a test note

# DESCRIPTION

foobar is a test.

I can use ual foobar to produce

FOOBAR(ual)                                                        FOOBAR(ual)



NAME
       foobar ‐ a test note

DESCRIPTION
       foobar is a test.

AUTHORS
       Gregory Anders.



                                 July 28, 2019                     FOOBAR(ual)

You can download ual here.

Tutorial: Booting Linux on the ZCU102

Wed, 13 Feb 2019 00:00:00 +0000

This post contains a step-by-step walk through on booting Linux on Xilinx’s ZCU102 MPSoC evaluation board.

Installing Linux on the Zynq MPSoC board is fairly straightforward if you take Xilinx’s advice and use their PetaLinux tool; however, I wanted to try my hand at getting a working Linux installation up and running without using PetaLinux, for a variety of reasons. Firstly, PetaLinux is massive. The initial download comes in close to 6 GB, with 10’s of GB more during the build process.

Second, PetaLinux is a nice process abstraction if you need to get something up quickly or if you don’t particularly care about knowing how the Linux boot process works, but that does not describe me. I didn’t want Xilinx to hold my hand and do everything for me. Understanding how things work is what makes a good engineer, so I wanted to at least get a working installation up on my own before resorting to a tool like PetaLinux.

It turns out that this process is a bit more involved than I would have assumed. Xilinx’s documentation in this area is scant and fragmented. The entire process is outlined in their wiki, but the information is often stale, badly formatted, obscure, or just plain wrong. This guide aspires to be what Xilinx’s wiki should be: a step-by-step solution from nothing to Linux.

What You’ll Need

A ZCU102 evaluation board
A USB-connection to the board’s UART (the kit comes with this cable)
An SD card with 2 partitions: a FAT32 boot partition, and an ext4 file system partition
Some kind of Linux installation (a virtual machine on Windows is fine)
A Vivado installation, including the Vivado SDK. You can run Vivado and the SDK on Windows if you like, but you must have SDK installed on Linux, even if you don’t use it. The SDK distribution contains all of the cross compilers and other tools required to build the Linux kernel and other components.

Step 1: Build your design in Vivado

This process is pretty well outlined in Vivado’s tutorials and in many other places, so I won’t go too in-depth here. Basically, create whatever design it is you want in Vivado (using the Zynq IP and any other IP you want to use) and create a bitstream. This part should be familiar to anyone who’s programmed an FPGA with Vivado before.

Once the bitstream is finished compiling, go to File > Export > Export Hardware. Make sure “Include bitstream” is checked.

Step 2: Launch SDK

While still in Vivado, go to File > Launch SDK. You can accept the defaults in the Launch SDK window or specify a custom location – it doesn’t really matter, but just put it somewhere easy to find because you’ll be digging files out of this location eventually.

Once in SDK you’ll have a hardware platform called <your_top_module>_hw_platform_0.hdf. The hardware platform is what contains all of the files to initialize the processing system (PS) to the settings you gave it in the IP integrator. The HDF file also contains your bitstream and any drivers for custom IP you might have included.

Step 3: Create the First Stage Boot Loader (FSBL)

In the Vivado SDK, go to File > New > Application Project. In the New Project dialog that appears, call the project fsbl and set the OS Platform to standalone. Make sure your hardware platform is selected in the Hardware Platform dropdown box, and also make sure psu_cortexa53_0 is selected in the Processor box. Don’t worry about any of the options in the Target Software group. Click Next. The next window is the Templates window. In the left pane, select Zynq MP FSBL (near the bottom). Finally, click Finish.

You can optionally enable debugging statements in the FSBL. In the Project Explorer, expand the newly created fsbl application and open the file xfsbl_config.h under the src directory. Enable debug printing by changing one of the following #define values to 1U. Each line enables more verbose output, i.e. enabling FSBL_DEBUG_DETAILED_VAL will print basically everything, while FSBL_PRINT_VAL will print much less. If you’re not sure, go ahead and enable FSBL_DEBUG_DETAILED_VAL, just in case:

#define FSBL_PRINT_VAL              (0U)
#define FSBL_DEBUG_VAL              (0U)
#define FSBL_DEBUG_INFO_VAL         (0U)
#define FSBL_DEBUG_DETAILED_VAL     (1U)

Step 4: Create the PMU Firmware (PMUFW)

This step is almost identical to the last one. Create a new Application Project and this time call it pmufw. Set the Processor to psu_pmu_0. Click Next and select the only option in the Templates window (ZynqMP PMU Firmware). Click Finish.

Step 5: Create the Device Tree Binary (dtb) files

First, clone Xilinx’s device-tree-xlnx repository (it doesn’t matter where). Once cloned, make sure you check out the tag corresponding to the Vivado version you are using. This is very important, and has potential to really trip you up down the road if you forget this step. For example, I built mine using Vivado 2018.2, so I checked out the tag xilinx-v2018.2 using

$ git checkout tags/xilinx-v2018.2

Once you’ve checked out the correct tag, go back to Vivado SDK and click Xilinx > Repositories in the menu bar. Click New… next to Global Repositories and add the device-tree-xlnx directory you just cloned. Click OK.

Go to File > New > Board Support Package. Near the bottom of the window you should see a box called “Board Support Package OS”. This box should now have an option called device_tree. Click that option and then click Finish. In the Board Support Package Settings window that comes up, click device_tree on the left and enter {BOARD zcu102-rev1.0} in the Value column of periph_type_overrides.

Finally, press Ctrl+B or click Project > Build All to build the FSBL, PMU Firmware, and device tree sources. You will need to compile the device tree sources into flattened blobs (.dtb) yourself. Many Linux distributions contain a tool called dtc that does this, or you can clone it from here and build it yourself. Whichever path you take, the command to use will be:

$ dtc -I dts -O dtb -o system.dtb system-top.dts

Note the change from system-top.dts to system.dtb. The top-level device tree source file from SDK is called system-top.dts, but the default U-Boot configuration Xilinx expects a device tree blob called system.dtb. This is an annoying gotcha.

In my case, SDK produced both a system-top.dts and a system.dts file. This appears to be due to a bug in Xilinx’s device tree generator (see here). The system.dts file contains only a single entry that sets the local MAC address of the Ethernet controller. You can either add the line

/include/ "system.dts"

to the top of your system-top.dts file or you can append the contents of system.dts to system-top.dts.

Once you have your system.dtb file, copy it to the FAT32 (boot) partition of your SD card.

Step 6: Compile U-Boot

Clone Xilinx’s U-Boot repository. Again, be sure to check out the tag corresponding to the version of Vivado you’re using. First, we need to set up our build environment:

$ source /path/to/Vivado/SDK/settings64.sh
$ export CROSS_COMPILE=aarch64-linux-gnu-

Note that you’ll need to repeat that for every new terminal window you open.

To build U-Boot, navigate to the U-Boot repository you cloned earlier and run the following:

$ make xilinx_zynqmp_zcu102_rev1_0_defconfig
$ make menuconfig
$ make

Once U-Boot is done compiling, you should see a u-boot.elf file in that directory.

Step 7: Compile Arm Trusted Firmware (ATF)

This process is similar to building U-Boot. Clone Xilinx’s ARM Trusted Firmware repository, check out the tag corresponding to your Vivado version, set the CROSS_COMPILE environment variable, and then run

$ make PLAT=zynqmp bl31

Once complete, you should have a bl31.elf file under build/zynqmp/release/bl31/.

Step 8: Create a boot image

This step is covered pretty well in Vivado’s Embedded Design Tutorial. See the chapter on Boot and Configuration. The only thing that tutorial leaves out is including your FPGA bitstream. The FPGA bitstream MUST be added before the ARM Trusted Firmware (bl31.elf). This is because when the FSBL prepares to load the bitstream, it loads it into the same memory region that the ATF is loaded. If the ATF is loaded before the bitstream, then the ATF will be overwritten.

Copy the BOOT.bin file you just created to the FAT32 (boot) partition of your SD card.

An even easier way to do this is to use the zynqmp-boot-apps tool. Simply clone the tool and follow a few simple instructions to generate both the BOOT.bin file and your system.dtb file:

$ cd ~
$ git clone https://github.com/gpanders/zynqmp-boot-apps
$ mkdir -p ~/zynqmp-boot-apps/build
$ cp /path/to/u-boot.elf ~/zynqmp-boot-apps/build
$ cp /path/to/bl31.elf ~/zynqmp-boot-apps/build
$ cp /path/to/your/hardware_description_file.hdf ~/zynqmp-boot-apps/
$ cd ~/zynqmp-boot-apps
$ source /opt/Xilinx/SDK/<version>/settings64.sh
$ make

If the FAT32 (boot) partition of your SD card is mounted to /media/sd, then you can install the boot files using

$ sudo make INSTALL_DIR=/media/sd install

Step 9: Compile the Linux kernel

Clone Xilinx’s fork of the Linux kernel and check out the tag corresponding to your Vivado version. Again, make sure your CROSS_COMPILE and ARCH environment variables are set to aarch64-linux-gnu- and arm64, respectively.

Run make xilinx_zynqmp_defconfig to generate a default config file for the ZCU102. You can optionally run make menuconfig (or make nconfig or make xconfig or any of the other config make targets) to configure the Linux kernel, but it’s fine to leave it at the default settings if you want.

Run make to start building the kernel. This step usually takes a while. Once complete, you will have a file called Image under arch/arm64/boot. The Image format is uncompressed and does not have a U-Boot header. If you want to boot this image using bootm in U-Boot, you must manually use the mkimage utility from U-Boot to wrap this image in a U-Boot header. However, U-Boot also has a booti command that allows us to use an uncompressed image with no header and this is, in fact, what the default bootcmd is set to use by the Xilinx U-Boot fork.

Copy the Image file to the FAT32 (boot) partition of your SD card.

Step 10: Find or create a file system

There are a thousand and one ways to procure a file system for Linux, so do whatever works for you. If you want just a minimal Debian or Ubuntu file system, you can download one from eewiki.

You can also use a tool like buildroot or Yocto Linux to generate a customized file system for your specific application.

If your development machine is running a Debian-based distribution (such as Ubuntu) and you also want to use a Debian-based distribution on your board, you can use a tool called debootstrap. To create a Debian-based root file system, use the process outlined here.

However you get it, install the file system to the ext4 partition of your SD card.

Step 12: Boot Linux

We’re finally there! Put your SD card into your device and make sure the boot pins are set to boot from SD card mode (see ZCU102 User Guide in the section titled “MPSoC Device Configuration”). Set up your serial console to listen to interface 0 (on Linux this is /dev/ttyUSB0, on Windows it’s Silicon Labs USB to UART Bridge: Interface 0) and turn on the device. You should see the FSBL debug statements going through each of the partitions of the BOOT.bin. The FSBL will first program the FPGA with the bitstream, then load the ARM Trusted Firmware, finally followed by U-Boot. If everything is done correctly, U-Boot should then start booting Linux (assuming you don’t interrupt the autoboot process).

In the 2019.1 version of Xilinx’s U-Boot fork the default $bootcmd has been changed. As of this update, this new default command does not work for me, but you can revert the $bootcmd to its old value by running the following from the U-Boot prompt:

ZynqMP> setenv bootcmd "run sdboot"
ZynqMP> saveenv

Note that using run sdboot is technically deprecated (you will see a message in the console saying as much), so this should only be a temporary stopgap.

And that’s it! From here, everything else is just configuration and the hardest part is out of the way.

Feel free to contact me if you have any questions or suggestions for improvement.

The Birthday Problem

Mon, 07 Dec 2015 00:00:00 +0000

The “Birthday Problem” is a classic in introductory probability theory, and it never fails to impress the newly initiated. The problem goes like this:

How many people do you need to have in one room for the probability that at least 2 people in that room share a birthday to be 50%?

The answer may surprise you. Before we get to the answer though, let’s go real briefly into the math behind it.

The probability that two people in a group of $N$ people share a birthday is the inverse of the statement that everybody in the room has a unique birthday. Both of these cannot be true, and if one of these statements is false, the other must be true. Therefore, if we find the probability of one of these statements (call that probability $p$ ), the probability of the other is $1 - p$ . Therefore, we can find the probability that at least 2 people share a birthday by finding the probability that everyone in that room has a unique birthday, and subtracting that number from 1. Following so far?

So what is the probability that in a room of $N$ people, everyone has a unique birthday? Let’s do an example where we have only 5 people. There are 366 possible birthdays (including February 29), so the first person in this group of 5 people can have any of those 366 birthdays, which we represent as a fraction:

\[ P(\text{Unique birthday with 1 person}) = \frac{366}{366} = 1\]

This can be read in English as “The probability that someone has a unique birthday in a group of 1 people is 100%”. This is obvious: if there is only one person, then they of course have a “unique” birthday, since there is no one else in the room to have the same birthday as them.

The 2nd person in the room can have any day as their birthday except the same day as the first person, so there are 365 options for Person 2’s birthday. Therefore, if we have 2 people, the probability that they have different birthdays is

\[ P(\text{Unique birthdays with 2 people}) = \frac{366}{366} \cdot \frac{365}{366} = 0.997\]

This pattern continues for every additional person. Therefore, in the event that we have 5 people in a room, the probability that they all have different birthdays is

\[ P(\text{Unique birthdays with 5 people}) = \frac{366}{366} \cdot \frac{365}{366} \cdot \frac{364}{366} \cdot \frac{363}{366} \cdot \frac{362}{366} = 0.973\]

Hopefully you’re beginning to see a pattern here.

Now, to find the probability that, in a room of 5 people, at least 2 of them have the same birthday (that is, there are not all unique birthdays), we simply subtract the number above from 1

\[ \begin{aligned} & P(\text{2 or more people have the same birthday}) \\ &= 1 - P(\text{Unique birthdays with 5 people}) = 0.027 \end{aligned}\]

As you might expect, this number is pretty small (about 2.7%). So how many people do we need to have in a room in order for there to be a 50% chance that at least 2 people share a birthday?

The answer is 23.

Surprising, I know. That seems way too low, and you’re probably a little incredulous. But the math checks out. In fact, it gets even stranger. At 23 people, there is (roughly) a 50% chance that 2 people share a birthday; by the time we get to 50 people the probability is 97%, and by the time we have a room full of 70 people it is a near certainty that at least 2 people will share a birthday (it is only a near certainty: in order to have a mathematical, 100% certainty that 2 people in a group will have the same birthday, you must have at least 367 people).

Looking at the graph below, you’ll see that the probability creeps up slowly until it gets to around 10 people, where it begins to climb rapidly and then taper off again around 40.

For the mathematically inclined, the formula used to calculate this is

\[ 1 - \frac{1}{366^N}\frac{366!}{(366-N)!}\]

So, next time you’re in a room with 23 or more people, test the math for yourself and see if anyone shares a birthday - and bonus points for you if you can casually drop the phrase “probability theory” into your conversation.

Make Your Mac Clean Itself

Wed, 11 Nov 2015 00:00:00 +0000

Update (6/18/2019): Since writing this post I have released a tool called pushbroom that should make all of this easier and a bit more robust.

The original solution in the post below is, frankly, a rather unsophisticated way of solving this problem. If you don’t want to use pushbroom, Thoughtbot has another solution that utilizes Apple’s launchd utility.

If you use the Internet ever, then you’ve probably downloaded a thing or two. Those downloaded files really start to add up over time, and unless you’re some kind of Internet hoarder, you probably don’t have any reason to hold on to them.

Go ahead and open Finder and look in your Downloads folder. Is it crazy full? That’s bad. File systems are like bedrooms - they’re much better when cleaned up and organized.

If you’re on Windows there’s a great program called Belvedere that takes care of this problem for you. But if you’re on a Mac, you’ll most likely have to pay money for a similar program, and that’s just no good. Instead, follow the quick and easy tutorial below to have your Mac automatically delete any file or folder that has been in your Downloads folder for more than 30 days.

First, open the TextEdit app and create a new file. Click Format > Make Plain Text.

Now copy the following to that file.

0  */6  *  *  * find $HOME/Downloads -mtime +30 -depth 1 -exec rm -rfv "{}" >> $HOME/Logs/clean_downloads_$(date +"\%Y\%m\%d").log \;

I know that command looks intimidating, but it’s actually pretty straight forward. This file is what’s called a “crontab”, which is short for “chronological table”, an old name given to a program that runs at a certain time (chronologically). The first five numbers/stars tell the program when to run, and that last long command (find $HOME/Downloads ...) tells the program what to do. In this case, we want the program to run at the 0th minute (0) every six hours (*/6) on every day, month, and day of the week (* * *). The command says to find every file in our Downloads folder that is older than 30 days, delete it, and then write the name of the deleted file to a new file called clean_downloads_<date>.log, so we know what’s actually been deleted.

Save the file in your home folder and call it .crontab (with the period at the beginning!) and make sure you uncheck the box that says If no extension is provided, use ".txt".

Once the file is saved, open the Terminal app (it’s the one that looks like a little computer screen) and enter the following:

mkdir -p ~/Logs
crontab ~/.crontab

And that’s it! Now, every six hours your computer will scan your downloads folder to find anything older than 30 days and delete it. So make sure you move anything important out of your Downloads folder and somewhere more appropriate!

Every time a file is deleted it is tracked in a log file called clean_downloads_<date>.log inside a folder called Logs in your home directory. They’re named depending on when they were run: for example, to see what was deleted on November 10, 2015, you would want to look at the file called clean_downloads_20151110.log, etc.