Comments (5)
@zkochan Should the --offline
in Dockerfile
be updated to --prefer-offline
at https://pnpm.io/cli/fetch because --offline
fails.
Also, this is in run.sh
of my Dockerfile
at the end & still downloads the dependencies each time:
run.sh
#!/bin/bash
set -e
# TODO: Only install dependencies for drizzle migration
echo "Installing production dependencies"
cd scripts
pnpm config set store-dir ~/.pnpm-store
pnpm fetch
pnpm install --prod --prefer-offline
cd ..
echo "Creating '/data/users.prod.sqlite' using bind volume mount"
pnpm run db:migrate:prod & PID=$!
# Wait for migration to finish
wait $PID
echo "Starting production server..."
node server.js & PID=$!
wait $PID
See this is the output. No reused deps:
Installing production dependencies
Importing packages to virtual store
Already up to date
Progress: resolved 1, reused 0, downloaded 0, added 0
Progress: resolved 42, reused 0, downloaded 14, added 11
Progress: resolved 42, reused 0, downloaded 40, added 40
Progress: resolved 42, reused 0, downloaded 42, added 42, done
.../node_modules/better-sqlite3 install$ prebuild-install || node-gyp rebuild --release
.../node_modules/better-sqlite3 install: Done
But there's some modules in here I guess:
/app $ du -sh ~/.pnpm-store/v3/
23.0M /home/node/.pnpm-store/v3/
Any ideas what's the problem?
Store doesn't work in Dockerfile
nor outside of it.
My current Dockerfile
& run.sh
(see above code-block) is a bit simplified.
Dockerfile
FROM node:20-alpine AS base
# mostly inspired from https://github.com/BretFisher/node-docker-good-defaults/blob/main/Dockerfile & https://github.com/remix-run/example-trellix/blob/main/Dockerfile
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
RUN corepack enable && corepack prepare [email protected] --activate
# set the store dir to a folder that is not in the project
RUN pnpm config set store-dir ~/.pnpm-store
RUN pnpm fetch
# 1. Install all dependencies including dev dependencies
FROM base AS deps
# Root user is implicit so you don't have to actually specify it. From https://stackoverflow.com/a/45553149/6141587
# USER root
USER node
# WORKDIR now sets correct permissions if you set USER first so `USER node` has permissions on `/app` directory
WORKDIR /app
# Install dependencies based on the preferred package manager
COPY --chown=node:node package.json pnpm-lock.yaml* ./
COPY --chown=node:node /src/app/db/migrations ./migrations
USER root
RUN pnpm install --frozen-lockfile --prefer-offline
# 2. Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps --chown=node:node /app/node_modules ./node_modules
COPY --chown=node:node . .
# This will do the trick, use the corresponding env file for each environment.
COPY --chown=node:node .env.production .env.production
# Copied from https://stackoverflow.com/a/69867550/6141587
USER root
# Give /data directory correct permissions otherwise WAL mode won't work. It means you can't have 2 users writing to the database at the same time without this line as *.sqlite-wal & *.sqlite-shm are automatically created & deleted when *.sqlite is busy.
RUN mkdir -p /data && chown -R node:node /data
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
RUN pnpm build
# 3. Production image, copy all the files and run next
FROM base AS runner
USER node
WORKDIR /app
EXPOSE 3000
ENV PORT 3000
ENV HOSTNAME '0.0.0.0'
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
COPY --from=builder --chown=node:node /app/public ./public
# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=node:node /app/.next/standalone ./
COPY --from=builder --chown=node:node /app/.next/static ./.next/static
# Move the drizzle directory to the runtime image
COPY --from=builder --chown=node:node /app/src/app/db/migrations ./migrations
# Move the run script and litestream config to the runtime image
COPY --from=builder --chown=node:node /app/scripts/drizzle-migrate.mjs ./scripts/drizzle-migrate.mjs
COPY --from=builder --chown=node:node /app/scripts/package.json ./scripts/package.json
COPY --from=builder --chown=node:node /app/scripts/pnpm-lock.yaml ./scripts/pnpm-lock.yaml
COPY --from=builder --chown=node:node /app/scripts/run.sh ./run.sh
RUN chmod +x run.sh
CMD ["sh", "run.sh"]
from pnpm.
I don't see where you are copying your pnpm-lock.yaml file into your container before running RUN pnpm fetch
. This would prevent the fetch
from pulling any deps.
from pnpm.
@indiejames Search for COPY --chown=node:node package.json pnpm-lock.yaml* ./
I'm definitely doing it after RUN pnpm fetch
but that's how I saw many examples.
But now that you said it, I searched for it & found many examples: https://sourcegraph.com/search?q=pnpm+fetch+file%3ADockerfile&patternType=standard&sm=0
Personally, I didn't understand how to use it & I just copy-pasted it lol.
Will this solve the bug?
Also, any recommended PNPM + Docker setup that is optimized?
from pnpm.
Bdw, is this okay?
I moved the whole pnpm cache thing inside the 2nd FROM in dev:
FROM node:20-alpine AS base
# mostly inspired from https://github.com/BretFisher/node-docker-good-defaults/blob/main/Dockerfile & https://github.com/remix-run/example-trellix/blob/main/Dockerfile
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
RUN corepack enable && corepack prepare [email protected] --activate
# 1. Install all dependencies including dev dependencies
FROM base AS dev
# Root user is implicit so you don't have to actually specify it. From https://stackoverflow.com/a/45553149/6141587
# USER root
USER node
# WORKDIR now sets correct permissions if you set USER first so `USER node` has permissions on `/app` directory
WORKDIR /app
# set the store dir to a folder that is not in the project
RUN pnpm config set store-dir ~/.pnpm-store
# Install dependencies based on the preferred package manager
COPY --chown=node:node package.json pnpm-lock.yaml ./
RUN pnpm fetch
COPY --chown=node:node . .
I'm new to dockerfile so Idk much about multi-stage builds & if the pnpm config & pnpm fetch should be stored at the top FROM or not? I'm confusing it with scope like Javascript.
from pnpm.
So thanks @indiejames, that kinda fixed the error. I updated my Dockerfile to this by adding the PNPM_HOME part from https://pnpm.io/docker:
FROM node:20-alpine AS base
# inspired from https://pnpm.io/docker
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
# mostly inspired from https://github.com/BretFisher/node-docker-good-defaults/blob/main/Dockerfile & https://github.com/remix-run/example-trellix/blob/main/Dockerfile
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
RUN corepack enable && corepack prepare [email protected] --activate
# 1. Install all dependencies including dev dependencies
FROM base AS dev
# Root user is implicit so you don't have to actually specify it. From https://stackoverflow.com/a/45553149/6141587
# USER root
USER node
# WORKDIR now sets correct permissions if you set USER first so `USER node` has permissions on `/app` directory
WORKDIR /app
# set the store dir to a folder that is not in the project
RUN pnpm config set store-dir ~/.pnpm-store
# Install dependencies based on the preferred package manager
COPY --chown=node:node package.json pnpm-lock.yaml ./
RUN pnpm fetch
COPY --chown=node:node . .
ENV NODE_ENV development
ENV NEXT_TELEMETRY_DISABLED 1
# Copied from https://stackoverflow.com/a/69867550/6141587
USER root
# Give /data directory correct permissions otherwise WAL mode won't work. It means you can't have 2 users writing to the database at the same time without this line as *.sqlite-wal & *.sqlite-shm are automatically created & deleted when *.sqlite is busy.
RUN mkdir -p /data && chown -R node:node /data
USER node
RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install --frozen-lockfile --recursive --prefer-offline
RUN pnpm install [email protected]
# 2. Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
# Inspired by https://github.com/vercel/next.js/discussions/36935
RUN mkdir -p /app/.next/cache && chown -R node:node /app/.next/cache
# Persist the next cache in a volume
VOLUME ["/app/.next/cache"]
COPY --from=dev --chown=node:node /app/node_modules ./node_modules
COPY --chown=node:node . .
# This will do the trick, use the corresponding env file for each environment.
COPY --chown=node:node .env* ./
# set to production otherwise it throws error ⚠ You are using a non-standard "NODE_ENV" value in your environment. This creates inconsistencies in the project and is strongly advised against. Read more: https://nextjs.org/docs/messages/non-standard-node-env
ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1
RUN pnpm build
# 3. Production image, copy all the files and run next
FROM base AS runner
USER node
WORKDIR /app
EXPOSE 3000
ENV PORT 3000
ENV HOSTNAME 0.0.0.0
ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1
COPY --from=builder --chown=node:node /app/.env* ./
COPY --from=builder --chown=node:node /app/public ./public
# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=node:node /app/.next/standalone ./
COPY --from=builder --chown=node:node /app/.next/static ./.next/static
# Move the drizzle directory to the runtime image
COPY --from=builder --chown=node:node /app/src/app/db/migrations ./migrations
# Move the litestream binary to the runtime image from the litestream image
# You can use a specific version of litestream by changing the tag
# COPY --from=litestream/litestream:0.3.13 /usr/local/bin/litestream /usr/local/bin/litestream
COPY --from=litestream/litestream:latest --chown=node:node /usr/local/bin/litestream /usr/local/bin/litestream
COPY --from=builder --chown=node:node /app/litestream.yml /etc/litestream.yml
COPY --from=builder --chown=node:node /app/scripts ./scripts
Lmk if anything can be improved here?
Also, I have a question: if I do docker compose build
, would it use dependencies from the /pnpm/store
during pnpm install
? Bcz this kinda contradicts with docker as it builds fresh builds. But I did try it & it built really fast which didn't happen before so I'm guessing its true.
Final question: do I actually need pnpm fetch
& pnpm install -r --offline
now that I have store cached with these 2 lines?
RUN pnpm config set store-dir ~/.pnpm-store
RUN --mount=type=cache,id=pnpm,target=~/.pnpm-store pnpm install --frozen-lockfile --recursive --prefer-offline
I removed this:
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
Final dockerfile:
FROM node:20-alpine AS base
# mostly inspired from https://github.com/BretFisher/node-docker-good-defaults/blob/main/Dockerfile & https://github.com/remix-run/example-trellix/blob/main/Dockerfile
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
RUN corepack enable && corepack prepare [email protected] --activate
# 1. Install all dependencies including dev dependencies
FROM base AS dev
# Root user is implicit so you don't have to actually specify it. From https://stackoverflow.com/a/45553149/6141587
# USER root
USER node
# WORKDIR now sets correct permissions if you set USER first so `USER node` has permissions on `/app` directory
WORKDIR /app
# set the store dir to a folder that is not in the project
RUN pnpm config set store-dir ~/.pnpm-store
# Install dependencies based on the preferred package manager
COPY --chown=node:node package.json pnpm-lock.yaml ./
RUN pnpm fetch
COPY --chown=node:node . .
ENV NODE_ENV development
ENV NEXT_TELEMETRY_DISABLED 1
# Copied from https://stackoverflow.com/a/69867550/6141587
USER root
# Give /data directory correct permissions otherwise WAL mode won't work. It means you can't have 2 users writing to the database at the same time without this line as *.sqlite-wal & *.sqlite-shm are automatically created & deleted when *.sqlite is busy.
RUN mkdir -p /data && chown -R node:node /data
USER node
RUN --mount=type=cache,id=pnpm,target=~/.pnpm-store pnpm install --frozen-lockfile --recursive --prefer-offline
RUN pnpm install [email protected]
# 2. Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
# Inspired by https://github.com/vercel/next.js/discussions/36935
RUN mkdir -p /app/.next/cache && chown -R node:node /app/.next/cache
# Persist the next cache in a volume
VOLUME ["/app/.next/cache"]
COPY --from=dev --chown=node:node /app/node_modules ./node_modules
COPY --chown=node:node . .
# This will do the trick, use the corresponding env file for each environment.
COPY --chown=node:node .env* ./
# set to production otherwise it throws error ⚠ You are using a non-standard "NODE_ENV" value in your environment. This creates inconsistencies in the project and is strongly advised against. Read more: https://nextjs.org/docs/messages/non-standard-node-env
ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1
RUN pnpm build
# 3. Production image, copy all the files and run next
FROM base AS runner
USER node
WORKDIR /app
EXPOSE 3000
ENV PORT 3000
ENV HOSTNAME 0.0.0.0
ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1
COPY --from=builder --chown=node:node /app/.env* ./
COPY --from=builder --chown=node:node /app/public ./public
# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=node:node /app/.next/standalone ./
COPY --from=builder --chown=node:node /app/.next/static ./.next/static
# Move the drizzle directory to the runtime image
COPY --from=builder --chown=node:node /app/src/app/db/migrations ./migrations
# Move the litestream binary to the runtime image from the litestream image
# You can use a specific version of litestream by changing the tag
# COPY --from=litestream/litestream:0.3.13 /usr/local/bin/litestream /usr/local/bin/litestream
COPY --from=litestream/litestream:latest --chown=node:node /usr/local/bin/litestream /usr/local/bin/litestream
COPY --from=builder --chown=node:node /app/litestream.yml /etc/litestream.yml
COPY --from=builder --chown=node:node /app/scripts ./scripts
@zkochan I think Docker docs need to be updated. Unfortunately, Idk enough about Docker to update it but the page can definitely have some improvements regarding pnpm fetch
use which I don't see there. Or atleast mention, if we need to use pnpm fetch
with --recursive --prefer-offline
with the cache or not, i.e, RUN pnpm config set store-dir ~/.pnpm-store
& RUN --mount=type=cache,id=pnpm,target=~/.pnpm-store pnpm install --frozen-lockfile --recursive --prefer-offline
. I'm confused which one is actually needed?
from pnpm.
Related Issues (20)
- PNP mode doesn't work with modules prefixed with 'node:' HOT 1
- Issues with mulitple version of drizzle-orm between different packages in workspace HOT 3
- Filtered install should work only on an up-to-date lockfile HOT 2
- Can not change the dependencies version in .pnpmfile.cjs HOT 5
- pnpm install --dir . fails when deps has npx in install script
- ajv 8.15 caused the problem, I'm not sure if the pnpm up report is a bug or not HOT 2
- peer dependency issue with pnpm deploy
- Can't use specifier for a snapshot dependency in the lock file HOT 13
- Updating to pnpm 9 cause `pnpm install` to look for a workspace dependency on npm HOT 3
- install script throws ETXTBSY ETXTBSY: text file is busy, copyfile '/tmp/tmp.rPcedmGtun/pnpm' -> '/home/capaj/.local/share/pnpm/pnpm'
- v9.2.0 `pnpm --filter=mypackage exec some-command` produces unexpected output HOT 2
- ERR_PNPM_NO_MATCHING_VERSION HOT 1
- Error installing `parcel` (ELIFECYCLE - exit code 126) HOT 3
- `pnpm fetch` fails if at least one injected workspace dependency exists in the monorepo
- Monorepo resolve error when using nextui. HOT 1
- `pnpm update` does not patch update `0.x` production dependencies HOT 2
- Option to ignore carets HOT 2
- Inconsistent install behaviour due to config variable namings in Turkish locale
- pnpm breaks federated modules HOT 1
- strict package manager HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pnpm.