Git Product home page Git Product logo

dev's Introduction

Node CI

Jane's Emporium (Monorepo) of Dev Tooling

Packages

Rationale

I have a number of "dev tooling" libraries/packages that I maintain for my own personal use, but having them as N different repositories were getting too onerous to maintain (e.g. you need to update N different packages, and if you wanted to change something that was common across the repos, yup, going into the different repos, making sure they're up to date, making the changes, and then pushing it, N different times).

Thus, I've crammed all of them into a monorepo to hopefully lower the burden of "doing things N times".

Notes about operating the monorepo

NPM settings

Lerna by default will use the top-level .npmrc file for its npm operations. You can override this by putting in a .npmrc in any of the package sub-repos.

Dependency Management/Workspaces

All package management/workspaces is delegated to npm. We use useWorkspaces: true to tell lerna to use npm for managing package scopes and the like.

Automated checks (git hooks)

Normally, in a non-monorepo setup, you'd install your dev dependencies (such as your eslint and prettier config, and in my case, skip-ci as well), and have eslint and prettier npm scripts set up to lint your codebase. And then, you'd have git hooks set up at the repo level, running said linters via lint-staged, which would look across all of the staged files and runs eslint/prettier, which read off of the root config. And then in CI, you run the same eslint/prettier commands from the root to see if people fucked around and pressed --no-verify.

However, in a monorepo, it's a bit trickier, because:

  1. You'd ideally like to make use of the "run the lint commands only for the files that changed" thing in CI (premature optimization, I know)
  2. You might be using different eslint/prettier configs across the various sub-repos.
  3. You also need to format (if not lint, depending on your repo setup) the root level, aside from all of the actual sub-repos.

First, let's talk about how to run the lint/format commands in a monorepo setting.

Now, lerna (or, if they've even bothered to update the documentation, NX) itself recommends that you just install and run all this linting shit at the root level (not sure how I feel about their recommendations extending to testing): https://lerna.js.org/docs/faq#root-packagejson.

Thinking about what would be the "ideal" workflow:

  • When you change something in a commit, we want that to be covered by the linter/formatter, and only the changed files (lint-staged covers this "for free", making sure we don't have to lint/format files that aren't affected).
  • When you run the checks in CI for a PR, you want to be checking the lint/format only for the files that changed in that PR.
  • When you run the checks in CI for the master branch (because let's be honest, we all just push to master sometimes), you want to be checking all of the files for lint/format, since we have no idea how many commits have been added to master in between the latest CI run and the previous one.

Thus, in this case, lerna's recommendation certainly checks out - we invoke lint-staged just as before (git hook), and setup CI (depending on the branch) to run the lint command from the root level, on only the files that changed.

Ok, now for how to deal with different configs across the various folders/sub-repos, this part is actually surprisingly easy and doesn't require changes in either our npm scripts at the root, or any of the lint-staged commands to lint/format the staged files.

In particular, eslint will automatically pick up the "nearest" (see https://eslint.org/docs/latest/user-guide/configuring/configuration-files#cascading-and-hierarchy for more info) configuration file, so you can drop in whatever config files you want for any of your sub-repos, and it'll automatically pick up the right config file to apply to the file you want to lint.

As for prettier, well, you really should only have the one at the top, so... yeah.

And, if you want to get even more fancy, you can add separate lint-staged configuration files for each sub-repo, as it, too, does the "look at the closest config file for the file I'm about to touch" thing as eslint. However, in most cases, just the config at the root level should suffice, as eslint and prettier commands don't need to differ across different sub-repos, as seen above.

Semantic Commit

To control all commit-based workflow for not only the various packages within this repo, but also the repo itself (i.e. the "top-level"), we expect all commits to follow the semantic commit pattern.

The exact config is based on the @janejeon/commitlint-config package (located within the commitlint-config/ folder), and we not only lint commit message before committing them via husky, but also check them in CI.

In particular, in CI, we want to check all commits for a branch to make sure that no "non-semantic" commits get through, which means having to pull down all commits when checkout out on git.

To automatically generate a commit message that adheres to the semantic commit ruleset, you can run npm run commit, which relies on commitzen, which relies on commitlint. The commit generation can be configured via https://commitlint.js.org/#/reference-prompt.

Publishing

All packages are tracked by lerna using semantic commits. Packages can be versioned and published via lerna.

To version and publish:

  1. Make sure you are on the master branch.
  2. Run lerna version to version packages according to the semantic versioning rules. This will update package.json versions, create git tags, and update CHANGELOG.md files.
  3. Run lerna publish from-git to publish the new versions to npm.
  4. Push the git tags with git push --follow-tags.

For now, this is being run manually while I get used to this workflow.

dev's People

Contributors

dependabot-preview[bot] avatar dependabot[bot] avatar janejeon avatar renovate-bot avatar renovate[bot] avatar

Watchers

 avatar  avatar

dev's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

git-submodules
.gitmodules
  • packages/skip-ci/repo master@4b01fbf57648adc7303d3762a934ead866470a58
github-actions
.github/workflows/ci.yml
  • actions/checkout v4
  • actions/setup-node v3
  • actions/checkout v4
  • actions/setup-node v3
  • actions/checkout v4
  • actions/setup-node v3
  • actions/checkout v4
  • actions/setup-node v3
.github/workflows/codeql-analysis.yml
  • actions/checkout v4
  • github/codeql-action v2
  • github/codeql-action v2
  • github/codeql-action v2
.github/workflows/ossar-analysis.yml
  • actions/checkout v4
  • github/ossar-action v1
  • github/codeql-action v2
npm
package.json
  • @vitest/coverage-istanbul ^0.34.2
  • @vitest/ui ^0.34.2
  • lerna ^8.0.2
  • lint-staged ^15.0.0
  • npm-run-all2 ^6.0.0
  • typescript ^5.1.6
  • vitest ^0.34.2
packages/commitlint-config/package.json
  • @commitlint/cli ^17
  • @commitlint/config-conventional ^17
  • @commitlint/config-lerna-scopes ^17
  • @commitlint/cz-commitlint ^17
packages/eslint-config-typescript/package.json
  • @typescript-eslint/eslint-plugin ^7.0.0
  • @typescript-eslint/parser ^7.0.0
packages/eslint-config/package.json
  • eslint ^8
  • eslint-config-prettier ^9.0.0
  • eslint-config-standard ^17
  • eslint-plugin-import ^2
  • eslint-plugin-n ^15
  • eslint-plugin-promise ^6
packages/got/package.json
  • @types/express ^4.17.17
  • express ^4.18.2
  • got ^13
  • got-scraping ^3
  • got-ssrf ^2
packages/prettier-config/package.json
  • prettier ^3
packages/renovate-config/package.json
packages/skip-ci/package.json
  • node >=14
packages/tsconfig/package.json
nvm
.nvmrc
  • node 20

  • Check this box to trigger a request for Renovate to run again on this repository

Split out "core" eslint-config

  • Make eslint, eslint-config-prettier, eslint-config-standard, eslint-plugin-import, eslint-plugin-n, and eslint-plugin-promise into peer dependencies
  • switch from node to n
  • Trim down the ESLint config down to just the above
  • eslint-plugin-import for regular javascript
  • extend plugin:promise/recommended
  • extend plugin:n/recommended
  • eslint-plugin-lodash plugin and extend recommended
  • eslint-plugin-markdown plugin and extend recommended
  • no-extend-native and no-use-extend-native
  • regexp
  • sonarjs
  • security
  • eslint-config-turbo for monorepo usage

Link all of the packages together

Instead of having different versions of packages cross-referencing each other installed in package.json, leverage lerna bootstrap to just "link" all of the packages together as a "*" dependency.

Create "frontend" eslint config

Have it as a separate package for peerDeps purposes

  • react
  • react-hooks
  • react-app
  • xss
  • rule for no class components?
  • eslint-plugin-import for react/jsx

Create dependency bundles

Probably split them out into "universal", "backend", and "frontend"

Some ideas for prod:

  • nopp: a small, universal package to prevent prototype pollution straight up?
  • trace and clarify: backend packages to be run on init to help with debugging - do we still need this w/ VS Code, given that we're never going to use this in production?

Dev dependencies:

  • husky v8
  • lint-staged
  • is-ci
  • skip-ci
  • eslint config
  • prettier config
  • commitlint config
  • commitlint-related tooling
  • jest-related tooling (jest, jest-junit, node-notifier)
  • typescript-related tooling (typescript, typescript-eslint, ts-jest)
  • cross-env
  • onchange
  • concurrently
  • npm-run-all
  • wait-on
  • rimraf
  • copyfiles

Setup renovate config for this repo

Basically, the idea is to have a renovate.json at the root repo, with something like

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["local>JaneJeon/renovate-config"]
}

pointing at the local JSON.

Revamp prettier-config

Split out prettier as peerDependencies, as they are now installed by default.

Make sure npm version is >=7 (which is when the "install peer deps by default" behaviour was added) via engines key.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.