My Quest for the Perfect TS Monorepo

Photo by Raphael Koh on Unsplash

I have spent an embarrassing amount of time this year trying to improve my tooling and understanding as it relates to monorepos. I will try to share some of this journey with you, so hopefully you don’t have to.

Advancements in tooling have elevated the JS/TS ecosystem tremendously over the past years, but you might be surprised at the amount of things you can still run into when you do not fully understand how everything connects.

Since both tooling and monorepos are pretty dry subjects, I will aim to keep things terse, and link external sources wherever I can, but it won’t be a short read. There is a lot to cover.

2024 Updates

This article was first published in December 2023. Since then, I have changed the text and the boilerplate code to include:

• Turbo v2 watch mode; this simplified some parts of the setup
• Typescript references
• A clean task
• A standardized typescript configuration

Working Example

If you want to jump straight in, here is an example codebase featuring a combination of a web app, two shared libraries and two backend services.

There are parallel branches for PNPM (main) and NPM. The readme also discusses some details that didn’t make it into this article.

I learned a lot from the excellent Turborepo starter and their “kitchen sink” example code, so I recommend to check that out and read their monorepo handbook.

This article goes quite a bit deeper, and attempts to combine pieces of information to give a clear overview of all the considerations and common pitfalls.

Some Background

For the past 7 years or so, I have dedicated most my professional time to two medium-size projects for two different companies, and both are built on the same platforms.

The stack itself is largely irrelevant to the issues and solutions discussed in this article, but for context it is:

• Node.js backend using Firebase and the Google Cloud Platform
• React.js clients based on Next.js and hosted by Vercel
• Firestore and Redis for data
• Typescript

The first of these projects started in September 2017. At the time, I had insufficient experience with monorepos, and the available tooling was not convincing, so I chose to avoid the added complexity and lump everything together.

The second project started in June 2019. By then, it was clear to me that monorepos are quite essential for full-stack projects where you deploy to multiple platforms, but I still wanted to keep things simple so I applied Yarn workspaces without much else.

Frustrations

The second setup felt alright for a while, but some aspects were clearly sub-optimal or even frustrating. I was manually managing the build order of packages and then regularly forgetting things or triggering redundant builds just to be sure everything was up-to-date.

When I later used a similar setup in a team context on a different project, colleagues regularly bumped into issues because they forgot to (re)build one of the packages.

As time went on, ESM adoption grew in the community, and we started running into dependency updates that were pure ESM. After some failed attempts at getting them to play nicely with the rest of our codebase, we ended up pinning some of those to older versions, which is not a sustainable solution.

The setup of the first project, with all of its dependencies for multiple platforms in a single package manifest, was not making ESM adoption any easier. In addition, it started to cause other issues like causing slow deployments and cloud functions cold-start times.

Working for two companies simultaneously is already challenging enough without having to waste time and energy on tooling and things that really should just work. In order to stay sane, I felt I had to try my best to understand and tackle these issues once and for all.

I set out to create the mono-ts boilerplate as a proof-of-concept, covering all the aspects that were essential to my projects. The boilerplate provides a working example of all the concepts discussed in this article, so please use that as a reference if things are unclear.

The Main Subjects

To me, the targets for a good monorepo setup are:

• Fast and deterministic build and task orchestration
• Consume and produce ESM where desired
• Deploy packages in isolation
• IDE go-to-definition for both code and types
• Live code updates / hot reloading during development

Deployment is not something all developers worry about, because it depends on the platforms used. In our case we use Firebase which currently does not support monorepos natively, and that turned out to be a major challenge.

Build and Task Orchestration

I was happy when Turborepo came out, because it solved my problems with build orchestration, yet it was simple to adopt and backed by Vercel.

It was not the first solution of its kind. Nx and Rush had been around for some time, so it is only fair to mention them here, and I assume there have been other efforts. Turborepo and Nx appear to be similar in a lot of respects, but since I have no real experience with Nx, I will not try to make a comparison.

In a nutshell, Turborepo works by configuring what packages depend on each other on a per-task basis like building, linting, and starting a dev server. You define the inputs and outputs required for each, so it can cache intermediate results and compute things only when the input parameters have changed.

In addition, Vercel optionally provides a shared cloud cache for Turborepo for your project, so that the tasks you run on your machine can in turn speed up the tasks on your colleagues machine or a CI pipeline. It is not hard to imagine how this could be beneficial for larger projects.

Internal Packages; To Build or Not to Build

The most common reason for choosing a monorepo setup is that you want to separate shared code and make it reusable in multiple apps and server deployments. It keeps dependencies clear and promotes the separation of concerns.

For example, you might have multiple web applications using the same set of UI components, or server and client code using sharing some types, utilities and business logic.

Often, you have no desire to publish these internal packages to NPM, because the source code is private to a company, and you only ever use it in the context of this particular repository.

When working with Typescript, there are two different patterns for linking these types of shared internal packages.

1. The Traditional Built-Package Approach

In this pattern, you build, and optionally bundle, your TS code to JS. The package manifest will point to the compiled JS output. This is no different from how you would develop a single open-source package if you want to publish to NPM or JSR.

Even though bundling technically is optional, I highly recommend it because it solves a lot of things, and bundlers have been getting insanely fast and easy to use anyway.

For simplicity, I am going to assume you will use a bundler for all of your built packages.

Benefits of this approach are:

• Each package is built separately, and a tool like Turborepo will be able to efficiently cache things.
• (Using a bundler) It makes it easy to work with Typescript path aliases. because the bundled output will not contain relative paths. The bundles only import from node_modules because your code is combined into single independent files.
• (Using a bundler) You can produce ESM modules without using the strict import rules that are required by the format like .js and /index.js suffixes. This could be helpful when you want a large existing CJS codebase to produce ESM output. The reason for this is similar to path aliases; your bundled output files will not contain any relative imports.
• (Using a bundler) You can simultaneously support different output formats like ESM and CommonJS. This might be essential if you publish your code for tools that do not fully support ESM yet (like the React Native Metro bundler)

Downsides of this approach are:

• Requires a bit more configuration and tooling
• If you use a bundler, you might have to jump through some hoops to get your output to map nicely to your source code and types for IDE go-to-definition.

I will come back to these points around ESM and bundling so bear with me.

2. The “Internal Packages” Approach

The term was coined by Jared Palmer from Turborepo. In this case you omit the build (and bundling) steps, and configure the package manifest to point directly to your Typescript source files.

Benefits of this approach are:

• Simple to follow, with hardly any configuration
• Out-of-the-box live code updates for environments that use a dev server like Next.js
• Out-of-the-box IDE go-to-definition for your source files and types

Downsides of this approach are:

• It is less efficient, as you leave type-checking, compiling and bundling up to the consuming context. For those shared packages, you can not make use of Typescript incremental builds, Turborepo can not cache their results, and so the build times will increase.
• If you want to use Typescript path aliases, you will have to configure a unique one for each package, otherwise the compiler gets confused. I love using aliases but I would consider them non-essential if the file structure of your shared package is not complex and deeply nested.
• You can not publish the package to NPM, because the manifest assumes Typescript and a bundler.

Some of these downsides are also discussed in the blog post from Turborepo

My Preference

For me, the benefits of bundling outweigh the benefits of not building. I also prefer the mental-model of treating each package as a self-contained unit.

There has been a lot of progress in the bundler space, and the Rust-based implementations are getting so insanely fast, that I suspect bundlers will become an integral part of tooling, and as an application developer you would not have to think about it.

The mono-ts boilerplate contains both approaches for demonstration purposes.

ES Modules

A lot of people in the Javascript ecosystem have been experiencing pain and frustration trying to transition from the CommonJS modules, as they were invented for Node.js, to the modern ESModules standard that was later designed to be part of Javascript.

Whatever your opinions are on ESM and the transition, I think we can all agree that the format is superior to CJS, and we should try to adopt it preferably sooner than later.

My personal frustrations mostly came from a lack of understanding, and in the end, I feel that integrating with modern tools is not that complicated once you understand some basic principles.

I hope the following gives a good summary. I wish I had read something similar a couple of years ago, as it could have saved me a lot of time and frustration. I did come across some seemingly good resources before, but still it took a long time for me to connect the dots.

Your Typescript Code Might Output CJS

This one seems obvious, but I think not everyone is aware of this. If the output of your TS build has require statements in it, that means the output target is not ESM.

It would be a mistake to assume that you are writing ESM compatible code because your Typescript sources use import/export statements.

I think the thing that trips most people up, is that your editor and compiler will allow you to import an ESM module without any warning, but the CJS output with Node will give you an ERR_REQUIRE_ESM error at runtime if the code is not converted.

CJS Cannot Import From ESM at the Top-Level

Because CJS modules are synchronous, and ESM modules are asynchronous, it is by definition not possible to import ESM modules directly at the top-level in CJS.

If you need to import an ESM module in CJS code, you will have to use a dynamic import, like so:

const someEsModule = await import from “some-es-module”

But because CJS modules can not have a top-level await statement, you can only do this from within another async function, like so:

async function useSomeModule() {
  const someEsModule = await import from "some-es-module";
  someEsModule.someFun()
}

As I mentioned before, bundlers and the Typescript compiler do not warn about these kinds of incompatibilities, and you will only get an error at runtime.

Note that Next.js has a transpilePackages setting that converts ESM code for you. I assume some other modern front-end frameworks have a similar feature, but I am not aware of any server frameworks that provide this convenience.

ESM Can Import from CJS

This should not be an issue, because you are importing a synchronous modules into an asynchronous context.

However, I did come across situations where the imported CJS module did not expose the default export as it was promised by its types, and I had to use .default to get there. This seems to be a rarity and I did not want to get to the bottom of it, so I will just leave it as a warning.

Dynamic Imports Might be Replaced by Your Build Tools

If you use dynamic imports as described above, and you compile to a CJS target, your tool chain might convert these imports to regular require statements, so you still might see a ERR_REQUIRE_ESM runtime error.

In the firebase-tools repository you can find a workaround for Webpack in dynamicImport.js so possibly you can do something similar for other bundlers if necessary.

JS Extensions on Relative Import Paths

ESM is stricter than CJS. It does not want to guess what file to import and for this reason you need to add a file extensions to all relative imports. It also does not resolve a directory to its index file.

In VSCode you can use the settings below to append extensions to auto imports:

"typescript.preferences.importModuleSpecifier": "shortest",
"javascript.preferences.importModuleSpecifierEnding": "js",

Personally, I prefer to use bundlers on every package which prevents me from having to write the file extensions and index paths for relative imports.

Using a .ts Suffix

If you think that using a .js extension on imports that point to Typescript code feels odd, you will be happy to know that there is an alternative. If you set moduleResolution to bundler it will allow you to use a .ts extensions instead. I think this was first popularized by Deno.

The idea is that bundlers will know how to interpret things and output valid ESM code from that. In the future, Typescript might just be the next iteration of Javascript, so it could even become the default way to import local module files.

But, as the configuration value suggests, you have to use a bundler. The Typescript compiler does not alter import paths when it writes its output, the same way it does not resolve path aliases for you.

For more info on this see this.

ESM All the Things

Because ESM is the future, and is easily compatible with CJS, but not the other way around, all packages in mono-ts are as ES Modules. Only a few individual configuration files are explicitly defined as CJS where necessary.

Deploying Packages in Isolation

Docker

For Docker images, Turborepo has a neat solution which copies and prunes a package and its internal dependencies so that it is more suitable for Docker and its use of caching layers.

Firebase

My projects deploy code to a platform that does not currently support monorepos. The Firebase Functions deploy command wants to upload a self-contained directory similar to an NPM package. It simply runs the install command and then tries to execute the entry point as declared in the manifest.

If your code lives in a monorepo and uses internally shared packages, those shared packages will not be found if you only bundle and deploy your firebase package, and this is not a trivial problem to solve.

People have been resorting to all kinds of hacks and scripts to work around it, but an elegant solution did not exist. I wrote a separate article explaining the challenges around Firebase and the generic solution I eventually developed for isolating monorepo packages.

Deploy to Firebase without the hacks

The isolate process takes the target package and its internal dependencies, and constructs a self-contained isolate directory with a pruned lockfile that is ready for deployment.

The isolated output contains only the files that would have been part of the packages as if they were to be published to NPM.

This also allows you to deploy to Firebase from multiple packages simultaneously! 💅

IDE Go-To-Definition

There are two important things you want from your IDE in a monorepo setup:

• When you click on imported code from a shared package, like a function or class, you want your editor to jump to the original Typescript file that defined it, not the build or bundle output.
• When you click on an imported type from a shared package, you want your editor to take you to the original type definition, not the d.ts output file that was generated by tsc or your bundler.

There are 3 ways you can achieve this:

1. You use the “internal packages” strategy described earlier, but as I mentioned, this is not what I would recommend.
2. You specify TS references in your tsconfig.json file. This way the compiler knows the relationships between your packages, and will jump to the source. The setting is also required for enabling TS incremental builds.
3. You generate .d.ts.map or “type definition map” files. If you publish packages to NPM, or privately for sharing in other codebases in your company, this is your only option to support go-to-definition.

Generating Type Definition Map Files

You can instruct tsup to generate type definitions, but those will be based on the bundled output, and that is not what we want. I suspect it is for this reason that tsup also does not provide an option to generate type definition map files.

The solution is to instruct tsup to only output the source and source-map files. Then, in an additional build step, we instruct tsc to output the type definitions and their map files to the same output directory.

For example: tsup && tsc --emitDeclarationOnly

The flag tells tsc to only emit type declarations and no Javascript code. If you then also set declaration and declarationMap to true in your tsconfig (or pass these as extra flags) you will get the desired output.

The type files will not match the bundled file structure, but mirror the original source structure. Luckily this is not a problem. An IDE will find all of the type files, and for each file there is the corresponding map to take the user back to the original source.

Live Code updates

As mentioned earlier, if you use the “internal packages” approach, you will likely get live code updates for any environment that runs a dev server out-of-the-box, but for packages that build or bundle to Javascript you will have to use some sort of watch task.

Turbo Watch Mode

I am happy that Turborepo introduced a watch mode in version 2, because it alleviates the biggest pain points I still had when I first published this article. You can check mono-ts to see it in action.

Firebase Emulators

With Firebase emulators you can run and test your code for Functions and Firestore etc without having to deploy things. If you use the isolate-package solution separately, you point the source field in firebase.json to the isolated output, but because the emulators use the same entry point, this breaks the live code updates you would normally get, and that is a major bummer.

To solve this, I have had to fork the firebase-tools package to integrate isolate to run as a function at the time of deployment. This way you run the emulators on the original source code as usual.

The fork doesn’t touch any functionality of firebase-tools, besides adding this extra pre-deploy step, so consider it safe to use in any project. I will try to keep it in sync by regularly pulling changes from upstream and publishing updates under the same versions.

Hopefully the Firebase team is willing to adopt my solution at some point, or find a different solution to the problem.

A Standardized Typescript Configuration

Since Typescript v5.5 it is possible to use the ${configDir} variable in the tsconfig files. This makes it possible to create truly reusable configurations.

I have used this to create @codecompose/typescript-config in an attempt to standardize everything across my codebases. It reduces the tsconfig.json files to only the extendbut, unfortunately, not all tools yet interpret the extended config properly:

• Next.js will demand the include property to exist, but this is not a big deal.
• TSUP will not understand the config at all if you ask it to generate type with the dtssetting. For this reason, mono-ts is using tsc to generate the type definitions, even though it also uses TS references.

Conclusion

I feel honored that you have made it this far, on such a dry subject, and with literally zero jokes and memes to spice things up.

I think it definitely beats having to figure this stuff out on your own, so I hope you have learned something valuable.

I will try to update this article over time, as I plan to do for the boilerplate code. If you have suggestions for essential improvements, please leave a comment or create a Github issue there.