Service mode about wireit HOT 38 CLOSED

Is there anything I can do to help here? This missing feature does make it harder to use wireit in any project that has a server, or its own build tool with build-in watchers.

It's my top priority for wireit, but lately I have had to prioritize other projects. I have a start on the implementation, but it's not quite ready to land yet. Sorry for the delay.

from wireit.

@aomarks can you share what you have, so I can take a stab at completing it?
Worst case scenario. you have a PR to deny ;)

Thanks! I'm actually working on this actively again now, hoping to land it in the next week or so.

from wireit.

I also think that having some declarative way of signifying when you know that a continuous process is fully started for whatever uses it as a dependency would be nice (in the cases mentioned above where you are chaining multiple continuous processes together).

I could imagine that and maybe a "readyMatch" field that takes a regex string to look for in stdout to indicate a server is ready to count the dependency as complete.

Good ideas! These could potentially be follow up features for the basic server support, but here are some ideas to explore first to let us achieve the same thing by composing regular scripts on top of a server script.

cc @justinfagnani @ZachHaber

Wait for a TCP port to be listened on

npm i -D wait-on

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "serve:ready": {
      "command": "wait-on tcp:8080",
      "dependencies": [
        "serve"
      ]
    },
    "serve": {
      "command": "node lib/server.js",
      "server": true,
      "dependencies": [
        "build:server",
        {
          "script": "build:assets",
          "restart": false
        }
      ]
  }
}

Wait for the server to emit some stdout

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "serve:ready": {
      "command": "tail -f -n+0 server.log | grep -q \"Server ready\"",
      "dependencies": [
        "serve"
      ]
    },
    "serve": {
      "command": "node lib/server.js | tee server.log",
      "server": true,
      "dependencies": [
        "build:server",
        {
          "script": "build:assets",
          "restart": false
        }
      ]
    }
  }
}

from wireit.

Services support is now released in 0.7.3!

Docs are at https://github.com/google/wireit/#services

Changes from the pre-releases are the addition of readyWhen (determines when a service is considered ready/serving) and cascade (determines which dependencies cause a service to restart).

@SanderElias @thejustinwalsh @michaelwarren1106 Thank you for beta testing, was a big help!

@SanderElias In my projects I've found services to be stable enough to get this release out, and I haven't reproduced your issue yet, but assuming you still have it, let's continue discussing in a new issue and get to the bottom of it. Currently my suspicion is that we should focus on why wireit can't shut down ng serve.

from wireit.

@aomarks I see you are getting close to having this completed
If you need someone to test-drive this, let me know! I have a couple of projects that would be good candidates.

Hey @SanderElias -- it is quite close, yes! I just published 0.7.3-pre.1 if you want to try it out. Expect some bugs, but hopefully the basic behavior should work. Docs are at https://github.com/google/wireit/blob/service/README.md#services

from wireit.

@SanderElias @thejustinwalsh I just published 0.7.3-pre.5 which has a fix that hopefully addresses the problem of scripts not shutting down/starting up in the right order -- and also of needing to restart less often in general. Let me know if you get a chance to try it out!

from wireit.

What's your thought for how to do this? Something like a "daemon": true flag?

I could imagine that and maybe a "readyMatch" field that takes a regex string to look for in stdout to indicate a server is ready to count the dependency as complete.

from wireit.

The correct answer is probably "wait until this is supported," but I just naively tried to workaround this:

  "scripts": {
    "start": "concurrently \"wireit\" \"dhost site --bind-all\"",
    "build": "wireit"
  },
  "wireit": {
    "start": {
      "dependencies": [
        "build"
      ]
    },
    "build": {
      "command": "rollup --config ./rollup.config.js",

and got yelled at:

[0] ❌ package.json:5:14 This command should just be "wireit", as this script is configured in the wireit section.
[0]         "start": "concurrently \"wireit\" \"dhost site --bind-all\"",
[0]                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I think I would suggest something like this for now:

  "scripts": {
    "start": "npm run build && concurrently \"npm run build watch\" \"dhost site --bind-all\"",
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "command": "rollup --config ./rollup.config.js",
    }
  }

So, you get an initial build, followed by simultaneously building continuously and starting your server.

from wireit.

@aomarks I tried again this morning. It seems a bit better, but I'm still experiencing a fair amount of freezes in my tools.
When this happens, it seems WiteIt also stalls an stops monitoring completely.

@SanderElias I found a couple bugs from my own testing which could be related to the ones you have been seeing. Released 0.7.3-pre.3 with those fixes. Note this also includes #238 which is fairly useful with services too.

from wireit.

I hit this today: I somewhat naively configured a server script as a dependency of another server script (Firestore emulator as a dependency of a file server, file server as a dependency of app server, etc).

I did not get an error though - I just saw no progress on the wireit output. The script I ran was depending on a build step and a server and the output just stopped after the build completed. I thought the build might have hung.

In this case there aren't strict requirements that one server finish starting before another starts, so I can just make them all sibling dependencies of a step that won't complete. I've had previous project where I might have needed to wait for some specific output from the server before considering it started though.

from wireit.

I did eventually run into a spot where I needed to start a server to run an integration test...

from wireit.

In addition to a server as a dependency, wireit should also expect a server as the end result of a command in general.

I was trying to set up wireit to work for running a server in watch mode for development, and the issue I ran into is that the watch mode doesn't start watching again until after the server is shut down, which doesn't work well for a dev environment.

For reference:

I've tried several different approaches at this point to this problem with varying degrees of success.

nodemon is probably the best overall still despite being slow. tsc-watch is faster, but fails to properly shut the server down if you save it twice in quick succession which then requires manual intervention. tsc's watch mode has no way to run a command after it, which makes it impossible to use.

I also really love the if-file-deleted feature as that solves a huge issue I've been frustrated with.

{
  "scripts": {
    "start:dev": "env-cmd -e dev node dist",
    "local:dev": "env-cmd --no-override -e dev,local npm run start",
    "local:prod": "env-cmd --nor-override -e prod,local npm run start",
    "start": "wireit",
    "ts": "wireit"
  },
  "wireit": {
    "start": {
      "command": "node dist",
      "files": [
        "dist/**/*.js"
      ],
      "dependencies": [
        "ts"
      ]
    },
    "ts": {
      "command": "tsc --build --pretty",
      "clean": "if-file-deleted",
      "files": [
        "src/**/*.ts",
        "tsconfig.json"
      ],
      "output": [
        "dist/**",
        ".tsbuildinfo"
      ]
    }
  },
}

from wireit.

I was trying to set up wireit to work for running a server in watch mode for development, and the issue I ran into is that the watch mode doesn't start watching again until after the server is shut down, which doesn't work well for a dev environment.

Yeah, this is a high priority issue.

In your case, would restarting your server on each watch iteration work?

I was also thinking that you often have some dependencies which require a restart, and some which don't. So maybe you declare like this:

"wireit": {
  "my-server": {
    "command": "node my-server.js",
    "server": true,
    "dependencies": [
      {
        "name": "build:server-implementation",
        "restart-on-change": true
      },
      {
        "name": "build:assets",
        "restart-on-change": false
      }
    ]
  }
}

I also really love the if-file-deleted feature as that solves a huge issue I've been frustrated with.

🙂

from wireit.

I feel like restart-on-change would be problematic and redundant with "files".

I.e. "files": [] would indicate that it shouldn't ever restart. "files": ["*"] would presumably indicate that it'd restart when anything changes.

Then the logic would work with whatever "server" settings we have to indicate it's a continuous process (I'm not sure the best term). I.e. "files": [] in the build:assets command would function by telling it to just re-use whatever output. However if build:assets was a continuous process, then it would just keep it running until its files changed, and an empty array would signify that it will never change.

And of course, if there's dependencies for a continuous process, it'd process them in that order, since the individual dependencies would be specifying their own files and dependencies, and if the files for a longer running process changes, then kill the process and restart it. I suppose that if "files" aren't declared at all, the ideal would to treat it as "if any dependencies indicate they changed" then restart the continuous process.

I could imagine that and maybe a "readyMatch" field that takes a regex string to look for in stdout to indicate a server is ready to count the dependency as complete.

I also think that having some declarative way of signifying when you know that a continuous process is fully started for whatever uses it as a dependency would be nice (in the cases mentioned above where you are chaining multiple continuous processes together).

from wireit.

I'm going to start working on this feature, and I just updated the description of this PR to describe how the feature will work.

from wireit.

The correct answer is probably "wait until this is supported," but I just naively tried to workaround this:

  "scripts": {
    "start": "concurrently \"wireit\" \"dhost site --bind-all\"",
    "build": "wireit"
  },
  "wireit": {
    "start": {
      "dependencies": [
        "build"
      ]
    },
    "build": {
      "command": "rollup --config ./rollup.config.js",

and got yelled at:

[0] ❌ package.json:5:14 This command should just be "wireit", as this script is configured in the wireit section.
[0]         "start": "concurrently \"wireit\" \"dhost site --bind-all\"",
[0]                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

from wireit.

I also just hit this, Working around it by nodemon. I don't like the nodemon solution, as it requires yet another terminal to be open.
Something to consider is other toolchains that have their own "watch&serve" options like Angulars ng serve.
while an option "server": true would work well in that scenario, the wording "server" might throw off some of the devs using such a tool

from wireit.

Have you considered how something like livereload/hot module replacement might work in wireit?

One common pattern is to use an environment variable (e.g. ROLLUP_WATCH) to decide whether or not to include a livereload client in the generated bundle. Without wireit, you might have something like this:

"start": "concurrently \"rollup --config ./rollup.config.js --watch\" \"firebase serve --host 0.0.0.0\"",
"build": "rollup --config ./rollup.config.js",

where "start" is effectively "build" + --watch. (I'm imagining a future where static file hosting is also managed by wireit.)

This makes me wonder:

• What is the recommendation for when to use an underlying tool's watch mode v.s. wireit's? In its current form, it seems that the underlying tool's ability to inject a dev-only snippet like livereload gives it an advantage.
• Should there be some conditionality in wireit, either by setting an env variable (WIREIT_WATCH), or by conditionally running/not running a rule depending on the presence of a watch flag?
• How does flagging work between wireit rules? Should there be the ability to include --watch in a dependency rule?
• Should the invocation assertion be relaxed, to allow:

    "start": "yarn run -TB wireit --watch"

  I've gone back and forth on this one, since it can perhaps also be modeled as:

    "start": "yarn run build --watch",
    "build": "yarn run -TB wireit"

from wireit.

Is there anything I can do to help here?
This missing feature does make it harder to use wireit in any project that has a server, or its own build tool with build-in watchers.

from wireit.

@aomarks can you share what you have, so I can take a stab at completing it?
Worst case scenario. you have a PR to deny ;)

from wireit.

@aomarks I see you are getting close to having this completed 🎉
If you need someone to test-drive this, let me know! I have a couple of projects that would be good candidates.

from wireit.

@aomarks I have been testing for a few days now. Mostly in angular projects.
Overall, it works reasonably well. There are some issues tho. The angular tools seem to "freeze" on a fairly regular base. This doesn't happen if I run them without wireit. Somehow it seems the processes are restarted before they are correctly/fully exited.

This occurred to me when I saw "errors" in my ngServe that happened because one of the libs was rebuilding.
I can't reproduce at will, but it happens every 5 to 10 iterations.
If there is any way I can create some login or anything that's useful to you, let me know.

from wireit.

@aomarks I have been testing for a few days now. Mostly in angular projects. Overall, it works reasonably well. There are some issues tho. The angular tools seem to "freeze" on a fairly regular base. This doesn't happen if I run them without wireit. Somehow it seems the processes are restarted before they are correctly/fully exited.

This occurred to me when I saw "errors" in my ngServe that happened because one of the libs was rebuilding. I can't reproduce at will, but it happens every 5 to 10 iterations. If there is any way I can create some login or anything that's useful to you, let me know.

Thank you for beta testing @SanderElias, big help! I just published 0.7.3-pre.2 that has a number of fixes, mostly related to error handling, which could be related to the issues you saw. Let me know if you get a chance to try it out!

from wireit.

@aomarks this is awesome- thank you! 0.7.3-pre.2 is working great for my project.

from wireit.

@aomarks I tried again this morning. It seems a bit better, but I'm still experiencing a fair amount of freezes in my tools.
When this happens, it seems WiteIt also stalls an stops monitoring completely.

from wireit.

I gave this a test drive and am experiencing an issue when a service is a dependency of another service under watch mode; the dependency service doesn't appear to exit cleanly when pressing ctrl+c.

Here is a slimmed-down example of my setup...

...
  "scripts": {
    "build:prisma": "wireit",
    "dev:frontend": "wireit",
    "dev:backend": "wireit",
    "prisma": "cd src-tauri && cargo prisma"
  },
  "wireit": {
    "dev:frontend": {
      "command": "vite",
      "service": true,
    },
    "dev:backend": {
      "command": "tauri dev --no-watch",
      "service": true,
      "files": [
        "src-tauri/src/**/*.rs",
        "src-tauri/build.rs",
        "src-tauri/tauri.conf.json"
      ],
      "packageLocks": [
        "Cargo.lock"
      ],
      "dependencies": [
        "build:prisma",
        "dev:frontend"
      ]
    },
    "build:prisma": {
      "command": "pnpm run prisma generate",
      "files": [
        "src-tauri/prisma/schema.prisma",
        "src-tauri/prisma/migrations/**/*"
      ],
      "output": [
        "src-tauri/src/db/mod.rs"
      ],
      "packageLocks": [
        "Cargo.lock"
      ]
    }
...

In the example above, if I terminate the dev:backend service (by quitting the gui application) and then press ctrl+c, everything appears to be getting cleaned up. However, If I press ctrl+c to shut down both without exiting the parent service when I attempt to spin up the dev:backend service again with pnpm run dev:backend --watch, the port that the dev:frontend service was using is still bound.

Running ps `lsof -ti:1420` reveals that the vite process is still running in this circumstance.

Additionally, I have noticed that when you press ctrl+c to kill the service in watch mode, you do not get any output about the services shutting down, whereas if the service terminates from user action, you get a nice feedback message about the event. It would be really slick if the sigterm gave output about the services shutting down for peace of mind.

I haven't tried to repro this with other services, so it very well could be something specific to my setup.
Thanks for all your hard work, this is looking really great!

from wireit.

Thank you @thejustinwalsh ! Can you confirm which version of wireit, and also which OS?

from wireit.

@aomarks I'm still running into issues. It is getting better, but it still errors out quite often.
I'm on a project that also has an Angular front. this fronend is being handled by this wireit:

    "start:sandbox": {
      "command": "ng serve sandbox --port 4201",
      "files": [
        "projects/sandbox/tsconfig.json",
        "projects/sandbox/src/**/*.ts"
      ],
      "service": true,
      "dependencies": [
        {
          "script": "build:feutils", "triggersRerun": true
        },
        {
          "script": "build:googlemaps", "triggersRerun": true
        }
      ]
    },

When using watch-mode, the problem is that the service is not properly exited before the recompilation of the dependencies starts, at least not on every run. Once the ting is stalled, I need to restart it entirely to get it going again.

EDIT: I'm on pre.3

from wireit.

Thank you @thejustinwalsh ! Can you confirm which version of wireit, and also which OS?

Just moved to 0.7.3-pre.3 and am running MacOS 12.6.1 (21G217)

from wireit.

Did you get the errors you described on pre3?

from wireit.

When using watch-mode, the problem is that the service is not properly exited before the recompilation of the dependencies starts, at least not on every run. Once the ting is stalled, I need to restart it entirely to get it going again.

Hey @SanderElias, thanks again for the bug reporting!

Why is it important in this example for the service to exit before compilation of the dependencies starts? What I would expect would happen given the current implementation and your config is this:

1. wireit starts
2. build:feutils and build:googlemaps run to completion
3. start:sandbox starts
4. "watching for file changes"
5. a file that affects build:feutils changes
6. build:feutils runs to completion
7. start:sandbox stops
8. start:sandbox starts

Are you saying that you expect step 7 to run before step 6? Why does this cause stalling?

from wireit.

@SanderElias @thejustinwalsh I found an issue that's probably the cause of these reports -- indeed in some cases a second version of a service can sometimes start up before waiting for the first one to shut down. Working on fix now!

from wireit.

Also, if you were using triggersRerun: false before, then note the setting has been renamed to cascade: false since 0.7.3-pre.5. Also note that true is the default value, so there's no need to set it if you do want a change in a dependency to trigger a restart. (See #520)

from wireit.

Did you get the errors you described on pre3?

Yes. As well as on pre1, same results with both preview builds. I did not try pre2 though.

from wireit.

Awesome I will check out pre5 and report back asap!

from wireit.

@aomarks

6. build:feutils runs to completion
7. start:sandbox stops
8. start:sandbox starts

Are you saying that you expect step 7 to run before step 6? Why does this cause stalling?

Yes. That is part of the problem. The start-sandbox uses ng serve. This has its own watch mode, and it really doesn't like it when one of its dependencies gets recompiled while it is running. Killing before compiling the dependencies does make sense in this case.
While thinking about it, I think it would be nice to have some control over this behavior, because it might depend on the tools that you are using and what makes more sense.
I'll give the next pre-version a run tomorrow, and will let yo know what happens.

from wireit.

Yes. That is part of the problem. The start-sandbox uses ng serve. This has its own watch mode, and it really doesn't like it when one of its dependencies gets recompiled while it is running. Killing before compiling the dependencies does make sense in this case.
While thinking about it, I think it would be nice to have some control over this behavior, because it might depend on the tools that you are using and what makes more sense.
I'll give the next pre-version a run tomorrow, and will let yo know what happens.

Interesting, thanks! I'm trying to reproduce this problem locally -- I've got ng serve set up with a sample Angular project, but can you give me a little more information about what kind of files get generated by build:feutils?

When using watch-mode, the problem is that the service is not pwhyroperly exited before the recompilation of the dependencies starts, at least not on every run. Once the ting is stalled, I need to restart it entirely to get it going again.

It seems like, even if it would be better to stop the service before building the dependency (e.g. to avoid some unnecessary churn, or to avoid some error messages), ng serve is still going to get automatically restarted afterwards -- so things should automatically get back to a good state eventually. So, I'm wondering what exactly is happening when ng serve is getting "stalled" -- do you mean that it stops responding to SIGINT, so the whole wireit process gets stuck?

from wireit.

@aomarks It is an angular CLI workspace.
The build:feUrils is building a lib by doing ng build --project @libs/feUtils.

In an angular project, you can add such a library with the command ng generate library @libs/feUtis

When it gets stalled, it just seems to freeze completely, and saving anything doesn't seem to trigger anything anymore.
When I ^c out, there is still a process running ng serve in my task-list, and I need to kill this manually.

from wireit.