My quest for the perfect TS monorepo

Photo by Raphael Koh on Unsplash

I have spent an embarrassing amount of time this year trying to improve my tooling and understanding as it relates to monorepos. I will try to share some of this journey with you, so hopefully you don’t have to.

Advancements in tooling have elevated the JS/TS ecosystem tremendously over the past years, but you might be surprised at the amount of things you can still run into when you do not fully understand how everything connects.

Since tooling and monorepos are both pretty dry subjects, I will aim to keep things terse, and link external sources wherever I can, but it won’t be a short read. There are a lot of things to cover.

Working Example

If you want to jump straight in, here are the goods. Note that there are parallel branches for PNPM (main), NPM, and both classic and modern versions of Yarn.

I learned a lot from the excellent Turborepo starter and their “kitchen sink” example code, so I recommend checking that out and also read their monorepo handbook if you are new to monorepos in general.

This article goes quite a bit deeper, and attempts to combine pieces of information to give a clear overview of all the considerations and common pitfalls.

Some Background

For the past 6 years or so, I have been dedicating most my professional time to two medium-size projects for two different companies, but both built on the same platforms.

The stack itself is largely irrelevant to the issues and solutions discussed in this article, but for context, it is:

• Node.js backend based on Firebase and the Google Cloud Platform
• React.js clients based on Next.js and hosted by Vercel
• Firestore and Redis for data
• Typescript

The first of these projects started in September 2017. At the time I had insufficient experience with monorepos, and the available tooling was not convincing to me, so I chose to avoid the added complexity and lump everything together.

The second project started in June 2019. By then it was clear to me that monorepos are quite essential for full-stack projects where you deploy to multiple platforms, but I still wanted to keep things simple so I applied Yarn workspaces without much else.

Frustrations

The second setup felt alright for a while, but some aspects were clearly sub-optimal or even frustrating. I was manually managing the build order of packages and then regularly forgetting things or triggering redundant builds just to be sure everything was up-to-date.

When I later used a similar setup in a team context on a different project, members regularly bumped into issues because they forgot to (re)build one of the packages.

As time went on, and ESM adoption grew in the community, we started running into dependency updates that were pure ESM. After some failed attempts at getting them to play nicely with the rest of our codebase, we ended up pinning some of those to older versions, which is not a sustainable solution.

The setup of the first project, with all of its dependencies for multiple platforms in a single package manifest, was not making ESM adoption any easier. In addition, it started to cause other issues like slower server deployments and cold-start times.

Working on two projects simultaneously is challenging enough without having to waste time and energy on tooling and things that really should just work. In order to stay sane, I felt I had to tackle these issues once and for all, and feel confident about how everything works and connects.

I set out to create the mono-ts boilerplate as a proof-of-concept, covering all the aspects that were essential to my projects. The boilerplate provides a working example of all the concepts discussed in this article, so please use that as a reference if things are unclear.

The Main Subjects

For me, the pillars of a good monorepo setup are:

• Fast and deterministic build and task orchestration
• Consume and produce ESM where desired
• The ability to deploy packages in isolation
• IDE go-to-definition for both code and types
• Live code updates in development

The challenges around deployment is not something that most developers encounter, as it largely depends on the platforms used. In our case it relates to Firebase and the fact that its tools currently do not support monorepos natively.

Build and Task Orchestration

When Turborepo came out, I immediately knew that it was something I had been longing for, but it wasn’t the first solution of its kind. Nx and Rush have been around for some time, so it is only fair to mention them here, and I assume there have been other efforts. Turborepo and Nx appear to be similar in a lot of respects, but since I have no real experience with Nx, I won’t go into details.

In a nutshell, Turborepo works by configuring what packages depend on each other on a per task basis like building, linting, and starting a dev server. You define the inputs and outputs required for each, so it can cache intermediate results and compute things only when the input parameters have changed.

In addition, Vercel optionally provides a shared cloud cache for Turborepo for your project, so that the tasks you run on your machine can in turn speed up the tasks on your colleagues machine or a CI pipeline. It is not hard to imagine how this could be very powerful for larger projects.

Internal Packages; To Build or Not To Build

The most common reason for choosing a monorepo setup is that you want to separate shared code and make it reusable in multiple apps and server deployments. It keeps dependencies clear and promotes the separation of concerns.

For example, you might have multiple web applications using the same set of UI components, or server and client code using some of the same types and business logic.

Typically you have no desire to publish these internal packages to NPM, because often the source code is private, and you only ever use it in the context of this repository.

When working with Typescript, you can choose between two different patterns for linking these types of shared internal packages.

1. The Traditional Built-Package Approach

In this pattern, you build, and optionally bundle, your TS code to JS. You define the package manifest so that it points to the compiled JS output. This is exactly how you would do things if you wish to publish a package to NPM.

Some benefits of this approach are:

• Each package is built separately, and a tool like Turborepo will be able to efficiently cache things.
• Makes it easy to work with Typescript path aliases, because (by default) a bundler removes them from the output. The output will only import from node_modules as your code is all combined into one or more independent files.
• You can produce ESM modules without using the strict import rules that are required by the format like .js and /index.js suffixes. This could be helpful when you want a large existing CJS codebase to produce ESM output. The reason for this is similar to path aliases; your bundled output files will not contain any relative imports.
• You can output different formats like ESM and CJS simultaneously. I think this is mainly useful if you use tools that do not support ESM yet, which is not very likely nowadays.

Some downsides of this approach are:

• Requires more configuration and possibly tooling
• If you use a bundler, you might have to jump through some hoops to get your output to map nicely to your source code and types for IDE go-to-definition.
• It requires more configuration if you want to have a watch task producing live code updates for your dev environment.

I will come back to these points around ESM, bundling and watch tasks, so bear with me.

2. The “Internal Packages” Approach

The term was coined by Jared Palmer from Turborepo. In this case you omit any build step, and configure the package manifest to point directly to your Typescript source files.

Some benefits of this approach are:

• Simple to follow, with hardly any configuration
• Out-of-the-box live code updates for environments that use a dev server like Next.js
• Out-of-the-box IDE go-to-definition for your source files and types

Some downsides of this approach are:

• Less efficient for larger repositories. As you leave the compilation and bundling for the consuming context, everything has to be rebuilt when some of the packages change. Turborepo can not cache intermediate results, and therefore build times will increase.
• If you want to use Typescript path aliases, you will have to configure a unique one for each package, otherwise the compiler gets confused. I love using aliases but I consider them non-essential for internal packages, because those file structures are usually not deeply nested.
• You can not publish the package to NPM, because the manifest assumes Typescript and a bundler.

Some of these downsides are also discussed in the blog post from Turborepo

My Preference

I currently prefer to build all of my internal packages, mainly because I like things to scale without issues. But, I also like to cut complexity where I can, so my mind could change on this. The boilerplate contains both approaches for testing.

In one project I use the built-package approach to also publish one internal package privately on NPM, to share it with other repositories within the same company.

ES Modules

A lot of people in the Javascript ecosystem have been experiencing pain and frustration trying to transition from the CommonJS modules, as they were invented for Node.js, to the modern ES Modules format that is now part of the Javascript standard.

Whatever your opinions are on ESM and the transition, I think we can all agree that the format is superior to CJS, and we should try to adopt it, preferably sooner than later.

My personal frustrations mostly came from a lack of understanding, and in the end, I feel like integration with modern tools is not that complicated once you understand some basic principles.

I wish I had read something similar a couple of years ago, as it would have saved me a lot of time and frustration. I had come across some seemingly good resources, but somehow it took a long time before things really clicked with me.

Your Typescript code might output CJS

This one seems obvious, but I think not everyone is aware of this. If the output of your TS build process has require statements in them, that means the output target is not ESM.

It would be a mistake to assume that you’re writing ESM compatible code because your Typescript sources use import/export statements.

Your editor and compiler will allow you to import an ESM module without warning, but your CJS output will give you an error at runtime.

CJS can not import from ESM at the top-level

Because CJS modules are synchronous, and ESM modules are asynchronous, it is by definition not possible to import ESM modules directly at the top-level in CJS.

If you need to import an ESM module in CJS code, you will have to use a dynamic import, like so:

const someEsModule = await import from “some-es-module”

But because CJS modules can not have a top-level await statement, you can only do this from within another async function, like so:

async function useSomeModule() {
  const someEsModule = await import from "some-es-module";
  someEsModule.someFun()
}

As I mentioned before, bundlers and the Typescript compiler do not warn about these kinds of incompatibilities, and you will only get an error at runtime.

Note that Next.js has a transpilePackages setting that converts ESM for you. I assume some other modern front-end frameworks have a similar feature, but I am not aware of any server frameworks that provide this convenience.

ESM can import from CJS

This should not be an issue, because you are importing a synchronous modules into an asynchronous context. The mental model is similar to how you would use async functions.

I have come across situations where the imported CJS module did not get me the default export, and I had to use .default, but this seems to be a rarity and I didn’t get to the bottom of it, so I will leave it as a warning for now.

Dynamic imports might be replaced by your build tools

If you use dynamic imports as described above, and you compile to a CJS target, your tool chain might convert these imports to regular require statements, so you still get something like an ERR_REQUIRE_ESM runtime error.

In the firebase-tools repository you can find a workaround for Webpack in dynamicImport.js so possibly you can do something similar for other bundlers if necessary.

JS extensions on relative import paths

ESM is stricter than CJS. It does not want to guess what file to import and for this reason you need to add a file extensions to all relative imports. It also doesn’t resolve a directory to its index file.

If you use a bundler you could instruct it to output ESM even if your TS code omits the extensions, but personally, I prefer to write my code as ESM now.

In VSCode I use the settings below to append extensions to imports, and since most of my imports are automatic, I usually do not append the extensions myself.

"typescript.preferences.importModuleSpecifier": "shortest",
"javascript.preferences.importModuleSpecifierEnding": "js",

Using a .ts suffix

If you think that using a .js extension on import paths that point to Typescript code in your editor is a bit odd, it should be noted that there is a solution. If you set moduleResolution to bundler it will allow you to use a .ts extensions instead.

The idea is that modern bundlers know how to interpret things correctly and output valid ESM code from that. In the future, Typescript might just be the next iteration of Javascript, so it could even become the normal way to import code.

But, as the configuration value suggests, you have to use a bundler. The Typescript compiler does not alter import paths when it writes its output, the same way it does not resolve path aliases for you.

For more info on this see this.

ESM All the Things

Because ESM is the future, and is easily compatible with CJS, but not the other way around, all packages in mono-ts are as ES Modules. Only a few individual files are explicitly pinned to CJS where necessary.

Deploying Packages in Isolation

Docker

For Docker images, Turborepo has a neat solution which copies and prunes a package and its internal dependencies so that it is more suitable for Docker and its use of caching layers.

Firebase

My projects deploy code to a platform that does not currently support monorepos. The Firebase Functions deploy command wants to upload a self-contained directory similar to an NPM package, to then run a package install and execute the entry point as declared in the manifest.

If your code lives in a monorepo and uses internally shared packages, that is not a trivial thing to do. People have been resorting to all kinds of hacks and scripts to work around it, but an elegant solution did not exist.

My projects are also large enough that I would prefer to split the Firebase deployments into individual services. This would improve code organization, and deployment and cold-start times.

I wrote a separate article explaining the challenges around Firebase and the generic solution I eventually created for isolating monorepo packages.

Deploy to Firebase without the hacks

The isolate process converts the target package into a new isolated package with its own root. It then copies over any internal dependencies and generates a dedicated lockfile. It is more a drastic form of pruning than the Turborepo method, in that it only outputs the files and structure that would have been part of the packages if they were to be published to NPM.

This also allows you to deploy to Firebase from multiple packages! 💅

IDE Go-To-Definition

There are two important things you want from your IDE in a monorepo setup:

1. When you click on imported code from a shared package, like a function or class, you want your editor to jump to the original Typescript file that defined it, not the build or bundle output.
2. When you click on an imported type from a shared package, you want your editor to take you to the original type definition, and not the d.ts output file that might have been generated by tsc or your bundler.

If you use the “internal packages” strategy described earlier, these things will work out-of-the-box, because the package manifest links directly to your source files.

If you build or bundle your packages, this can be achieved via source-map files .js.map and type-definition-map files .d.ts.map

Getting there might require a some extra jumping through hoops, depending on your bundler.

At the moment I use tsup. It seems to be the best suited for my needs, but at the time of writing, it is not capable of generating all of this by itself.

The Problem

You can instruct tsup to generate .d.ts files, but these files will be based on the bundled output, and I assume it is for this reason that tsup does not provide an option to generate .d.ts.map files.

A Solution

In order to get the output we want, we can instruct tsup to only output the source map files, besides the bundled sources. We can then use tsc to output the type definitions and their map file to the same output directory.

The instruction looks something like tsup && tsc --emitDeclarationOnly

The flag tells tsc to only emit type declarations and no Javascript code. If you then also set declaration and declarationMap to true in your tsconfig (or pass these as extra flags) you will get the desired output.

The type files will not match the bundled file structure, but still mirror the original source structure. Luckily this is not a problem for your editor. It can find all of the type files, and for each file there is the corresponding map to take you back to the original type definition in your source code.

Live code updates

As mentioned earlier, if you use the “internal packages” approach, you will likely get live code updates for any environment that runs a dev server out-of-the-box, but for packages that build or bundle to Javascript you will have to use some sort of watch task.

Unfortunately, at the moment, Turborepo does not yet include a mechanism for watching file changes, but I will talk about two ways you could approach this.

Using parallel watch tasks for bundled packages

As described in the previous section, your bundled internal packages might require two separate commands to generate the Javascript and type declarations with their map files.

It is not possible to run these in series, one after another, because once you add a watch task to the first command, it will never finish executing, so we will have to run both in parallel.

The simplest solution I know uses npm-run-all, and your package manifest scripts could look like this:

"scripts": {
    "bundle": "tsup-node",
    "bundle:watch": "tsup-node --watch",
    "type:gen": "tsc --emitDeclarationOnly",
    "type:gen:watch": "tsc --emitDeclarationOnly --watch",
    "type:check": "tsc --noEmit",
    "build": "run-p bundle type:gen",
    "dev": "run-p bundle:watch type:gen:watch",
    "clean": "del dist tsconfig.tsbuildinfo",
    "test": "vitest",
    "coverage": "vitest run --coverage ",
    "lint": "eslint \"**/*.ts*\""
  },

The run-p command is an alias for npm-run-all --parallel

I think the strategy above can work reliably if you are willing to start the individual dev tasks manually. Each package dev task is persistent (because it doesn’t end) and so in your turbo.json file you can not define that one package’s dev task depends on another.

When you trigger the top-level dev task, it will start all dev tasks at the same time, so I assume that depending on the current build state of your packages, this might or might not lead to issues. It certainly doesn’t feel very deterministic to me.

For my projects this approach seems to be sufficient at the moment.

Turbowatch

The npm-run-all approach is clearly more of a workaround than a solid solution, so it might not be sufficient for complex monorepos.

Turbowatch was designed to fill the void of Turborepo’s missing watch mode. I have yet to try it for myself, so I can not really comment on it, but I encourage you to check it out. It can be used in tandem with Turborepo, and also on its own.

Firebase Emulators

With Firebase emulators you can run and test your code for Functions and Firestore etc without having to deploy things. If you use the isolate-package solution separately, you point the source field in firebase.json to the isolated output, but because the emulators use the same entry point, this breaks the live code updates you would normally get, and that is a major bummer.

To solve this, I have had to fork the firebase-tools package to integrate isolate to run as a function at the time of deployment. This way you run the emulators on the original source code as usual.

The fork doesn’t really touch any functionality of firebase-tools, besides adding this pre-deploy step, so I consider it safe to use in any project. I will keep it in sync by regularly pulling changes from upstream and publishing updates under the same versions.

Hopefully the Firebase team is willing to adopt my solution, or they will find a better way to support monorepos.

Conclusion

I feel honored that you have made it this far, on such a dry subject, and with literally zero jokes and memes to spice things up.

I think it definitely beats having to figure this stuff out on your own, so I hope you have learned something valuable.

I will try to update this article over time, as I plan to do for the boilerplate code. If you have suggestions for essential improvements, please leave a comment or create a Github issue there.