The great gatsby upgrade

[Fourth in a series of posts about GatsbyJS. All three initial posts have been updated: 1. Introduction and The SPA Calculation. 2. Tips for productionizing. 3. First notes on this blog’s source code.]

In the past year I’ve upgraded this blog’s version of Gatsby not just once, but twice. And wow, it has gotten so much better. And so has my blog. Fork it and play around! Or, if you’re not yet convinced, read on…

A powerful architecture

I could talk about any number of compelling Gatsby features, but the real game-changer for Gatsby is its deep extensibility. In the abstract, it is a system that surfaces data via self-documenting, type-safe GraphQL to consuming code, with many built-in primitives for assembling the resultant high-performance web application.

It really is that abstract, because you need to add ‘transformers’ to Gatsby project to supply the data - there’s nothing to start! For example, this blog uses gatsby-transformer-filesystem to pull files from disk, gatsby-transformer-remark to turn any markdown files found into HTML, and gatsby-transformer-sharp to process large images into various responsive sizes.

With that data, my code then generates pages and other assets like RSS files. But not before tweaking the GraphQL a little as an optimization - a perfect sample of the deep extensibility available in the system.

The ecosystem

A common pitfall of highly-customizable systems is that they are not accessible to the beginner. The most basic ‘hello world’ requires too many decisions, and the hunt through all the settings can be very lengthy.

Enter gatsby plugins. Experts can tackle that comprehensive extensibility and package it up for everyone else to use! Some plugins are very simple, and some are quite complex indeed.

The gatsby site reports over 2800 searchable plugins as of this writing. It was plenty for me! When updating my blog, things I had to do manually before became as simple as a search, then adding the plugin. No need to manually add and integrate catch-links - just add gatsby-plugin-catch-links to your gatsby-config.js!

The struggle

I won’t lie to you. Despite these improvements, or perhaps because of them, it was not an easy process upgrading from 0.12.7 to 3.13.0. Gatsby had changed a whole lot! I kept it simple: my goal for the upgrade was to keep the user experience exactly the same. No redesign rabbit holes. But it was still a major effort.

I had big plans in May 2020 of using quarantine time to upgrade to Gatsby 2. I got pretty far, but then stalled out due to Storybook/Gatsby incompatibilities. And, of course, the other challenges inherent in trying to establish new routines in the face of sweeping societal change. I bought a rowing machine and played a lot of Magic The Gathering Arena (more about both of these in future posts).

I came back in November of that year, started gaining momentum during the course of that month, and finished it all off the week of Thanksgiving. I made almost 60 commits that month! My first successful Vercel deployment was November 23rd 2020. Then there was another burst of activity in December to improve performance and turn on Javascript in a branch.

Now, as fall starts, almost a year later, I just upgraded to Gatsby 3.13.0.

Why Gatsby?

I first started using Gatsby in January 2016, and here we are in 2021. This is the longest I’ve stayed with one blogging platform. It had value for me as a React developer back then, but now it has a large collection of value propositions:

• Advanced handling of images, sharp-based preprocessing - it takes a lot of work to prepare an image for true responsiveness, given all the screen types out there today. Photoshop automation or ImageMagick scripts can help, but it’s a whole lot of bookkeeping to manage it all. Gatsby uses sharp and a lot of performance tuning expertise to make your images look great, all within its comprehensive build step.
• High-performance defaults - Performance is a key focus of the Gatsby ecosystem, and it means that a gatsby project does advanced things automatically: base64-encoded minimal representation of images along with other types of progressive enhancement, code-splitting, and prefetching. A fast initial load and fast subsequent clicks too.
• Intelligent file generation for proper caching - Javascript, images, styles, and other assets are all named based on the hash of their contents. No cache busters needed; all of these files can be cached for a long time. Your public endpoints and the page-data.json files representing them can’t be cached long-term, and that’s it. My caching configuration for Vercel is quite simple.
• Self-documentation via GraphQL - A lot of data is surfaced by Gatsby in GraphQL as soon as you add a few plugins, but it’s never confusing. The excellent GraphiQL exploration interface is available during gatsby develop, guiding you towards the queries you need to accomplish your goals.
• A push towards best practices - To be in the Gatsby ecosystem means that you are already using a lot of key best practices for web development, and both the documentation and community will continue in that path. For example, it’s very easy to turn your Gatsby site into a Progressive Web Application, if you choose. Sometimes you don’t choose, and are instead pushed: Gatsby 3.0 changed CSS module exports to ES modules because it was the current state of the art.
• React and Webpack under the covers - These widely-used technologies give you the power, even without writing any Gatsby plugins, to make a Gatsby project a full-blown application or a simple static site. And I’m quite familiar with them, so I’m in my element.

But I couldn’t stop here. I had to go deeper to really make it mine.

My customizations

Things have changed a lot since late 2016, the last time I made a large change to the technology of my blog. The industry has changed. I’ve changed. I now have a new set of requirements for code I’ll be living with for a while.

• Typescript - Happily, Gatsby 3.0 made this easier: gatsby-plugin-typescript plugin is now included by default. But it’s still a bit of a pain to go 100% Typescript for a Gatsby project. Your core gatsby-node.js and gatsby-config.js files must be Javascript - I’ve done a few backflips to keep them typed. And I’ve also added a separate script for type-checking, since that isn’t done during a build. Next, I’d like to get type-checked GraphQL.
• Storybook - My initial hurdle was with Gatsby’s <Link /> component. At the time, I wrote “Nothing found in my many google searches - people talking about it in the past, but none of their fixes work.” The key, months later, was to dig in and understand how I needed to tweak Storybook’s Webpack config to make things work. And I’m sure that some bugfixes landed too. But I got there. It’s so nice to add a permutation whenever I find a style bug.
• SCSS - Influenced by the Gatsby defaults in the past, I was using Less for my blog. I never really liked it. I’m glad to be back to a familiar CSS preprocessor.
• CSS Modules with generated d.ts files - Gatsby does push you down the CSS modules path, but if I was going to use a more advanced system like this, I wanted typings! It required a lot of work initially, but with Gatsby 3.0 it just requires a new plugin: gatsby-plugin-dts-css-modules. CSS modules are great - once you have them, you can do really cool things like minify your class names!
• GitHub Actions and Yarn offline cache - It’s a much simpler and much faster continuous integration system than I had before!
• Prettier - A simple addition to the project, it keeps everything tidy.

You can see the full set of recent changes in the project’s change log. These are just the highlights!

A contentious customization

I’ve previously spoken about my decision to disable Javascript on my blog. And this time around I did a good amount of work to maintain that decision in the face of a substantial amount of Gatsby change.

But it’s not just about the theory - let’s allow for the visceral sense of actually navigating with Gatsby’s Javascript and without!

This blog has no Javascript (probably where you’re reading this): https://blog.scottnonnenberg.com

And you can try out the Javascript-enabled version of this blog here, right now: https://blog-js.scottnonnenberg.com

What do you think?

Giving Javascript a chance

The Javascript-enabled site is pretty fast, and most people tell me that they can’t tell the difference between Javascript enabled and not. Funny enough, that says to me that downloading all that extra Javascript probably isn’t worth it!

I really did give it a good-faith effort, though. I did a bunch of work to take the download size down, both the bundle sizes and the page-data.json files.

The very first thing I did was install the excellent gatsby-plugin-webpack-bundle-analyser-v2 gatsby plugin to help me zero in on problem areas. The obvious next step was to remove lodash, moment, and underscore.string from the bundle, implementing myself the few functions I needed.

At that point it was time to look at what the browser was downloading. Spending time in the Network tab of the Chrome Dev Tools, I noticed that the page-data.json files for pages like index, popular and tags were quite large. Again, pretty obvious when I dug in - the React components shouldn’t be interacting with HTML at all, because that means all the HTML, for all the pages, has to be shipped to the client!

Now it started getting complex. I needed to think like I was writing a gatsby-transformer-xxx plugin - how might I surface just the information needed for each of my components? And the answer, like everything in Gatsby, is “expose it via GraphQL!” This is where the createResolvers Gatsby API comes in:

  createResolvers: ({ createResolvers }: CreateResolversArgs): any => {
    const resolvers = {
      MarkdownRemark: {
        htmlPreview: {
          type: 'String',
          resolve: async (source: PostType, args: any, context: any, info: any) => {
            const htmlField = info.schema.getType('MarkdownRemark').getFields()['html'];
            const html = await htmlField.resolve(source, args, context, info);

            const slug = source?.frontmatter?.path;
            if (!slug) {
              throw new Error(`source was missing path: ${JSON.stringify(source)}`);
            }
            return getHTMLPreview(html, slug);
          },
        },

This Typescript adds a new GraphQL field htmlPreview alongside the html field gatsby-transformer-remark already provides for our markdown files. The key tricky bit is that info.schema.getType and ultimate resolve() call - with those tortured lines we’re querying that sibling html field in the GraphQL to get what we need to construct our custom result. The full change adds another textPreview field and updates the consuming GraphQL queries.

It was a really big step forward, and I was finally starting to get comfortable in this environment!

I made a couple more commits to further slim down my React components and I was set. A few principles:

• Make sure the full html field is only fetched when displaying the whole post - everywhere else is some sort of pared-down preview
• Fetch htmlPreview/textPreview fields only when necessary. For the index page, only the first few show HTML previews, the next set are plaintext, and the final set don’t have any preview. We can express that right in the query.
• Use the full power of GraphQL, with parameters like sort, skip and limit to fetch minimal data. You can reach into frontmatter data right in a GraphQL sort/filter clause!

It’s a far cry from the Gatsby of old, putting all of your posts in that single bundle. But it’s still not good enough for my use case. I think the exercise was useful, despite that. And just maybe you’ll find my blog’s /js branch a better starting point!

Turning off Javascript

My very first steps towards the Gatsby 2.0 upgrade included gatsby-plugin-no-javascript, because I wasn’t about to have a core behavior of my blog go away with this so-called upgrade! It was quick and easy to install. At least the spirit of my original noProductionJavascript option was still around!

Once I had everything basically working post-upgrade, I started looking at the site more closely. I quickly discovered an errant 85kb Javascript polyfill file still included in my built pages. Disappointed that my plugin missed this, I pulled out one of my favorite quick-and-dirty tools: patch-package. It shortens the fork/fix feedback loop into ‘change the node module on disk and run patch-package’ - no git URLs in your package.json, no new fork in your GitHub profile, no fuss. I eliminated the unnecessary download with a quick patch.

From there I continued to find things that my original no-Javascript plugin wasn’t eliminating. I continued to patch, but I also found another highly-useful plugin: gatsby-plugin-no-javascript-utils. It allowed me to drop inline styles for a separate CSS file I could include from all of my pages, caching aggressively. Now I was really in business!

Later, I decided to source the author image at the bottom of my pages from local assets instead of a separate CDN domain, and in so doing give it the same treatment all other images had on the site: a base64-encoded thumbnail, different sizes available for different screens, etc. Enter gatsby-image. It was the recommended way to make images responsive, so I jumped in. But then I realized that it didn’t work without Javascript! It put some code into a <noscript> tag, so it was ready for browsers with Javascript disabled, but not site builders excluding it entirely! I was shocked, because my images in my markdown-generated posts had worked so well, Javascript or no. Exit gatsby-image.

But gatsby-image didn’t go far, really. As you do in this kind of situation, I replicated much of it for my author image. This is probably where I felt the most like an outsider in the Gatsby ecosystem, doing something the system wasn’t really meant to do. But it works!

The High and the Low

I definitely had my high points and low points during the process of refreshing my Gatsby knowledge and modernizing my blog.

The good:

• The ecosystem is great - you can find plugins that do most things, or at least good inspirations.
• GraphQL is powerful, typed, and self-documenting.
• The modern Gatsby API gives you a lot of extension points. I love that I can easily generate RSS/Atom in the build step without extra scripts.

The bad:

• It was a little difficult to turn off Javascript - but at least this time I didn’t have to pull request Gatsby itself!
• Storybook was quite difficult to set up when I first tried!
• Typescript still isn’t supported perfectly - I want a gatsby-config.ts!
• Major version upgrades are still painful, even from 2.x to 3.x.
  • Some key plugins didn’t make the major version jump. In some cases that’s just warnings on the console, and in other cases it’s a full crash. patch-package to the rescue once more!
  • It was hard to figure out why the promised v2 to v3 CSS modules change wasn’t working. Feels like I just happened to discover that it was because I still had gatsby-plugin-typescript manually installed.
  • I’m getting The "fixed" and "fluid" resolvers are now deprecated warnings, which tells me that I’m going to need to go back to gatsby-image soon. Maybe it’s now ready for no-Javascript builders like me?
• There were some unfortunate breaking changes:
  • With a new markdown parser came new sensitivities to certain nested formatting operators.
  • With a new plugin providing header anchor links, special characters were processed differently. Some deep links into my posts were broken.
  • My blog previously used Typography.js, but it hasn’t been updated lately and its API had changed too radically. So I ejected, using its generated CSS but none of its functionality.

Fork away!

I’ve done a bunch of work to modernize my blog, and I like working with it much more than I used to. It’s been a rewarding effort.

But it’s not just about me. With my readme and inline comments, it could be useful for you too! Fork it and add new storybook stories and React components! Start with the /js branch and add interactivity to the pages! Use it to express yourself, then deploy it to the world!