Comments (19)
Thanks @eduardoboucas.
Unfortunately, I haven't had any luck. I just tried this on "@netlify/plugin-nextjs": "3.4.2", but I'm either not accessing .jsons correctly or it's not working.
netlify.toml
[functions] node_bundler = "esbuild" included_files = ["json/collections/*.json"]
.json & getStaticProps
My .jsons live on
/public/collections/*.json
. I'm then trying to read the .json on the collection's getStaticProps with something like (pseudo-code):
const collection = await fs.readFile(path.join(process.cwd(), `/json/collections/${params.handle}.json`), 'utf8')
ENOENT: no such file or directory, open '/var/task/json/collections/tires.json'
Console logging
process.cwd()
still returns/var/task
, so I get nothing backNotes
I inspected the
/out_functions/next_collections_handle.js
and I don't see any references to these json files.Any idea what I may be doing wrong?
P.S reading from the filesystem works fine on every context except (isPreview === true && NODE_ENV === 'production')
Hi @juanpprieto.
The path should be relative to the project root, so if the files live under /public
, I believe the config should be:
[functions]
node_bundler = "esbuild"
included_files = ["public/json/collections/*.json"]
And to read them you can try:
const collection = await fs.readFile(path.resolve(`public/json/collections/${params.handle}.json`), 'utf8')
Let me know if this helps.
from next-runtime.
@MejanH hey! we pushed up a temporary solution for this supported in the plugin and wanted to let you know since you were the very first to surface this issue! i will note that this is a temporary, unstable solution. we're also working on a greater, platform-wide solution for this type of file inclusion. if you'd like access to the short term next-specific plugin solution, feel free to email me [email protected] or dm on twitter @levlinds. otherwise, i'll leave this issue open until the longer term solution rolls out! <3
from next-runtime.
the platform solution is finally out!!! https://docs.netlify.com/configure-builds/file-based-configuration/#functions
please go ahead and give it a shot!
from next-runtime.
Game changer! 🔥🔥🔥
Thanks! @eduardoboucas
from next-runtime.
Thanks for the prompt response @lindsaylevine!
I've got this up and running now. Essentially I found that process.cwd()
didn't work on Netlify (but did work in local development, and on Vercel. For more context, the previous version of getServerSideProps
I was using accessed the filesystem in a manner something like the below (I have inlined a helper function here for ease of reference):
import matter from "gray-matter";
export const getServerSideProps = async ({ params }) => {
const fs = require("fs");
const path = require("path");
const directoryPath = path.join(process.cwd(), "_posts");
const pageSlugs = fs.readdirSync(directoryPath);
const allPosts = pageSlugs.map((slug) => {
const fullPath = path.join(directoryPath, slug);
const fileContents = fs.readFileSync(fullPath, "utf8");
const document = matter(fileContents);
return { slug, ...document };
});
return {
props: { ...params, allPosts },
};
};
The revised version that does work across localhost, Netlify and Vercel borrows heavily from this repository: https://github.com/cassidoo/next-netlify-blog-starter and looks more like the below, where the getPosts
method is now pretty much identical to this: https://github.com/cassidoo/next-netlify-blog-starter/blob/master/utils/getPosts.js
import matter from "gray-matter";
import getPosts from "path/to/utils/file";
export const getServerSideProps = async ({ params }) => {
const allPosts = ((context) => {
return getPosts(context);
})(require.context("../../posts", true, /\.md$/));
return {
props: { ...params, allPosts },
};
};
For that code to work you also need to run yarn add raw-loader
and include this setting in your next.config.js
:
config.module.rules.push({
test: /\.md$/,
use: "raw-loader",
});
It would be handy to have process.cwd()
set to the project root like it is in localhost and Vercel to avoid always having to include a hard-coded string reference like "../../posts" or "../posts" or whatever, but not a big deal. Always open to ideas if people know of ways to make that reference more dynamic while staying on Netlify. At the moment process.cwd() === '/var/task'
in the Netlify functions created for Server-Side Rendered (SSR) pages
Anyway, mostly just sharing in case these notes help anyone else (and also as a plug for all the excellent Netlify start repos that exist like the one linked in this comment!) 😄
PS: might also be worth posting the good news about the latest platform solution for this problem here too: https://answers.netlify.com/t/how-to-access-files-from-nextjs-api-function/35347/2
from next-runtime.
@coreyaus hey corey! per @eduardoboucas aka the engineer behind included_files
, this is something the team is working on (the process.cwd() stuff)!
thanks so much for sharing your notes regardless, we appreciate it and i'm sure other users will too!!
i also left a comment on that community thread 👍
from next-runtime.
@juanpprieto hey there! my apologies for not circling back here!!! my teammate @eduardoboucas just recently shed some light on that question here. per his most recent comment, it's something we'll be rolling out soon. you can subscribe to #359 to see when eduardo gives that release update!
from next-runtime.
@MejanH ok WEW finally got this deployed and diagnosed! so here's what's going on:
because these pages are trying to use ISR (see revalidate
flag in your getStaticProps
functions), our plugin turns these pages into netlify functions. from within the function['s node env], the files (aka the markdown) you're trying to read "from disk" are not bundled with those functions, so it's failing to find them and erroring out.
this is something we can/should address, but for now, if you want this to Just Work, don't use ISR/revalidate. it will work as a plain SSG'd/prerendered page.
thanks for opening that separate repo for me and let me know if you have any other questions!! i'm going to change the title if that's cool to be a little more specific to the issue :)
from next-runtime.
Thanks for the help. Will wait for this feature.
from next-runtime.
hey @MejanH thanks for opening this! happy to help. however, need a little bit more info from you. the fastest/easiest way for me to help you is for you to open a repo that reproduces this issue and that i can deploy myself. from the deploy logs you shared, i don't see a markdown dir under pages which means, wherever you have it, it's excluded from what next-on-netlify or next as a framework would handle. let me know!
from next-runtime.
Thanks for the reply. For the demo, I just followed the next.js learn pages. At build time there is no problem with 'fs". But I don't why it's not working on view.
from next-runtime.
hey, i spent a bit of time trying to get typescript-final
working on my machine. that repo seems somewhat broken. when you initially run next dev
, you get an invalid date error. then next build
fails trying to look for files ending in .md.html
. because this doesn't even work locally for me, there's not much i can do.
also, i can tell from your logs in your original post that what you have deviates from typescript-final
in at least some way. can you please open a repo with exactly what you tried to deploy to netlify?
from next-runtime.
Hi, I have created a repo as you said. tried with npm run build
and next build
as this official post says. Still the same error. It gets published successfully but not working.
7:50:06 PM: Build ready to start
7:50:08 PM: build-image version: 53b83b6bede2920f236b25b6f5a95334320dc849
7:50:08 PM: build-image tag: v3.6.0
7:50:08 PM: buildbot version: cdcf5e6d038c4a5c0bbc618d40b7eb21a2893d7d
7:50:08 PM: Building without cache
7:50:08 PM: Starting to prepare the repo for build
7:50:08 PM: No cached dependencies found. Cloning fresh repo
7:50:08 PM: git clone https://github.com/MejanH/nextjs-blog-incremental-static
7:50:09 PM: Preparing Git Reference refs/heads/main
7:50:11 PM: Starting build script
7:50:11 PM: Installing dependencies
7:50:11 PM: Python version set to 2.7
7:50:13 PM: v12.18.0 is already installed.
7:50:13 PM: Now using node v12.18.0 (npm v6.14.4)
7:50:13 PM: Started restoring cached build plugins
7:50:13 PM: Finished restoring cached build plugins
7:50:14 PM: Attempting ruby version 2.7.1, read from environment
7:50:16 PM: Using ruby version 2.7.1
7:50:16 PM: Using PHP version 5.6
7:50:16 PM: Started restoring cached node modules
7:50:16 PM: Finished restoring cached node modules
7:50:16 PM: Installing NPM modules using NPM version 6.14.4
7:50:32 PM: > [email protected] install /opt/build/repo/node_modules/sharp
7:50:32 PM: > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
7:50:34 PM: info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.10.0/libvips-8.10.0-linux-x64.tar.br
7:50:37 PM: > @ampproject/[email protected] postinstall /opt/build/repo/node_modules/@ampproject/toolbox-optimizer
7:50:37 PM: > node lib/warmup.js
7:50:38 PM: AMP OPTIMIZER Downloaded latest AMP runtime data.
7:50:38 PM: npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/watchpack-chokidar2/node_modules/fsevents):
7:50:38 PM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
7:50:38 PM: npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
7:50:38 PM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
7:50:38 PM: added 714 packages from 467 contributors and audited 716 packages in 20.898s
7:50:39 PM: 81 packages are looking for funding
7:50:39 PM: run `npm fund` for details
7:50:39 PM: found 2 low severity vulnerabilities
7:50:39 PM: run `npm audit fix` to fix them, or `npm audit` for details
7:50:39 PM: NPM modules installed
7:50:39 PM: Started restoring cached go cache
7:50:39 PM: Finished restoring cached go cache
7:50:39 PM: go version go1.14.4 linux/amd64
7:50:39 PM: go version go1.14.4 linux/amd64
7:50:39 PM: Installing missing commands
7:50:39 PM: Verify run directory
7:50:42 PM:
7:50:42 PM: ────────────────────────────────────────────────────────────────
7:50:42 PM: Netlify Build
7:50:42 PM: ────────────────────────────────────────────────────────────────
7:50:42 PM:
7:50:42 PM: ❯ Version
7:50:42 PM: @netlify/build 8.0.0
7:50:42 PM:
7:50:42 PM: ❯ Flags
7:50:42 PM: deployId: 5fd8bf0e213e5d0054b2fb66
7:50:42 PM: mode: buildbot
7:50:42 PM:
7:50:42 PM: ❯ Current directory
7:50:42 PM: /opt/build/repo
7:50:42 PM:
7:50:42 PM: ❯ Config file
7:50:42 PM: No config file was defined: using default values.
7:50:42 PM:
7:50:42 PM: ❯ Context
7:50:42 PM: production
7:50:42 PM:
7:50:42 PM: ❯ Installing plugins
7:50:42 PM: - @netlify/[email protected]
7:51:21 PM:
7:51:21 PM: ❯ Loading plugins
7:51:21 PM: - @netlify/[email protected] from Netlify app
7:51:22 PM:
7:51:22 PM: ────────────────────────────────────────────────────────────────
7:51:22 PM: 1. onPreBuild command from @netlify/plugin-nextjs
7:51:22 PM: ────────────────────────────────────────────────────────────────
7:51:22 PM:
7:51:23 PM: ** Adding next.config.js with target set to 'serverless' **
7:51:23 PM:
7:51:23 PM: (@netlify/plugin-nextjs onPreBuild completed in 14ms)
7:51:23 PM:
7:51:23 PM: ────────────────────────────────────────────────────────────────
7:51:23 PM: 2. Build command from Netlify app
7:51:23 PM: ────────────────────────────────────────────────────────────────
7:51:23 PM:
7:51:23 PM: $ next build
7:51:24 PM: warn - No build cache found. Please configure build caching for faster rebuilds. Read more: https://err.sh/next.js/no-cache
7:51:24 PM: info - Creating an optimized production build...
7:51:43 PM: warn - Compiled with warnings
7:51:43 PM: ./node_modules/next/dist/next-server/server/load-components.js
7:51:43 PM: Critical dependency: the request of a dependency is an expression
7:51:43 PM: ./node_modules/next/dist/next-server/server/load-components.js
7:51:43 PM: Critical dependency: the request of a dependency is an expression
7:51:43 PM: ./node_modules/next/dist/next-server/server/require.js
7:51:43 PM: Critical dependency: the request of a dependency is an expression
7:51:43 PM: ./node_modules/next/dist/next-server/server/require.js
7:51:43 PM: Critical dependency: the request of a dependency is an expression
7:51:43 PM: ./node_modules/next/dist/next-server/server/require.js
7:51:43 PM: Critical dependency: the request of a dependency is an expression
7:51:43 PM: ./node_modules/node-fetch/lib/index.js
7:51:43 PM: Module not found: Can't resolve 'encoding' in '/opt/build/repo/node_modules/node-fetch/lib'
7:51:43 PM: info - Collecting page data...
7:51:44 PM: info - Generating static pages (0/5)
7:51:45 PM: info - Generating static pages (1/5)
7:51:45 PM: info - Generating static pages (2/5)
7:51:45 PM: info - Generating static pages (3/5)
7:51:45 PM: info - Generating static pages (5/5)
7:51:45 PM: info - Finalizing page optimization...
7:51:46 PM: Page Size First Load JS
7:51:46 PM: ┌ ● / (ISR: 1 Seconds) 631 B 71.3 kB
7:51:46 PM: ├ /_app 0 B 60.2 kB
7:51:46 PM: ├ ○ /404 2.75 kB 62.9 kB
7:51:46 PM: ├ λ /api/hello 0 B 60.2 kB
7:51:46 PM: └ ● /posts/[id] 433 B 71.1 kB
7:51:46 PM: ├ /posts/pre-rendering
7:51:46 PM: ├ /posts/ssg-ssr
7:51:46 PM: └ /posts/testing-ssg
7:51:46 PM: + First Load JS shared by all 60.2 kB
7:51:46 PM: ├ chunks/commons.87066d.js 13 kB
7:51:46 PM: ├ chunks/framework.9ec1f7.js 39.9 kB
7:51:46 PM: ├ chunks/main.bb7417.js 6.22 kB
7:51:46 PM: ├ chunks/pages/_app.ec7c07.js 296 B
7:51:46 PM: ├ chunks/webpack.e06743.js 751 B
7:51:46 PM: └ css/2fd5d76e8788e1d11cbb.css 264 B
7:51:46 PM: λ (Lambda) server-side renders at runtime (uses getInitialProps or getServerSideProps)
7:51:46 PM: ○ (Static) automatically rendered as static HTML (uses no initial props)
7:51:46 PM: ● (SSG) automatically generated as static HTML + JSON (uses getStaticProps)
7:51:46 PM: (ISR) incremental static regeneration (uses revalidate in getStaticProps)
7:51:46 PM:
7:51:46 PM: (build.command completed in 23.1s)
7:51:46 PM:
7:51:46 PM: ────────────────────────────────────────────────────────────────
7:51:46 PM: 3. onBuild command from @netlify/plugin-nextjs
7:51:46 PM: ────────────────────────────────────────────────────────────────
7:51:46 PM:
7:51:46 PM: ** Running Next on Netlify package **
7:51:46 PM: 🚀 Next on Netlify 🚀
7:51:46 PM: Functions directory: netlify-automatic-functions
7:51:46 PM: Publish directory: /opt/build/repo
7:51:46 PM: Make sure these are set in your netlify.toml file.
7:51:46 PM: 🌍️ Copying public/ folder to /opt/build/repo
7:51:46 PM: 💼 Copying static NextJS assets to /opt/build/repo
7:51:46 PM: 💫 Setting up API endpoints as Netlify Functions in netlify-automatic-functions
7:51:46 PM: 💫 Setting up pages with getInitialProps as Netlify Functions in netlify-automatic-functions
7:51:46 PM: 💫 Setting up pages with getServerSideProps as Netlify Functions in netlify-automatic-functions
7:51:46 PM: 🔥 Copying pre-rendered pages with getStaticProps and JSON data to /opt/build/repo
7:51:46 PM: 💫 Setting up pages with getStaticProps and fallback: true as Netlify Functions in netlify-automatic-functions
7:51:46 PM: 💫 Setting up pages with getStaticProps and revalidation interval as Netlify Functions in netlify-automatic-functions
7:51:46 PM: 🔥 Copying pre-rendered pages without props to /opt/build/repo
7:51:46 PM: 🔀 Setting up redirects
7:51:46 PM:
7:51:46 PM: (@netlify/plugin-nextjs onBuild completed in 127ms)
7:51:46 PM:
7:51:46 PM: ────────────────────────────────────────────────────────────────
7:51:46 PM: 4. Functions bundling
7:51:46 PM: ────────────────────────────────────────────────────────────────
7:51:46 PM:
7:51:46 PM: Packaging Functions from netlify-automatic-functions directory:
7:51:46 PM: - next_api_hello/next_api_hello.js
7:51:46 PM: - next_index/next_index.js
7:51:46 PM: - next_posts_id/next_posts_id.js
7:52:00 PM:
7:52:00 PM: (Functions bundling completed in 14.3s)
7:52:00 PM:
7:52:00 PM: ────────────────────────────────────────────────────────────────
7:52:00 PM: Netlify Build Complete
7:52:00 PM: ────────────────────────────────────────────────────────────────
7:52:00 PM:
7:52:00 PM: (Netlify Build completed in 1m 18.6s)
7:52:01 PM: Caching artifacts
7:52:01 PM: Started saving node modules
7:52:01 PM: Finished saving node modules
7:52:01 PM: Started saving build plugins
7:52:01 PM: Finished saving build plugins
7:52:01 PM: Started saving pip cache
7:52:01 PM: Finished saving pip cache
7:52:01 PM: Started saving emacs cask dependencies
7:52:01 PM: Finished saving emacs cask dependencies
7:52:01 PM: Started saving maven dependencies
7:52:01 PM: Finished saving maven dependencies
7:52:01 PM: Started saving boot dependencies
7:52:01 PM: Finished saving boot dependencies
7:52:01 PM: Started saving rust rustup cache
7:52:01 PM: Finished saving rust rustup cache
7:52:01 PM: Started saving rust cargo bin cache
7:52:01 PM: Finished saving rust cargo bin cache
7:52:01 PM: Started saving go dependencies
7:52:01 PM: Finished saving go dependencies
7:52:04 PM: Build script success
7:52:04 PM: Starting to deploy site from ''
7:52:04 PM: Creating deploy tree
7:52:04 PM: Creating deploy upload records
7:52:05 PM: 5 new files to upload
7:52:05 PM: 3 new functions to upload
7:52:06 PM: Starting post processing
7:52:06 PM: Post processing - HTML
7:52:07 PM: Post processing - header rules
7:52:07 PM: Post processing - redirect rules
7:52:07 PM: Post processing done
7:52:07 PM: Site is live ✨
7:52:28 PM: Finished processing build request in 2m19.653681844s
JSON error after publishing
{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "Error: ENOENT: no such file or directory, scandir '/var/task/posts'",
"trace": [
"Runtime.UnhandledPromiseRejection: Error: ENOENT: no such file or directory, scandir '/var/task/posts'",
" at process.<anonymous> (/var/runtime/index.js:35:15)",
" at process.emit (events.js:314:20)",
" at processPromiseRejections (internal/process/promises.js:209:33)",
" at processTicksAndRejections (internal/process/task_queues.js:98:32)"
]
}
from next-runtime.
@MejanH hey! we pushed up a temporary solution for this supported in the plugin and wanted to let you know since you were the very first to surface this issue! i will note that this is a temporary, unstable solution. we're also working on a greater, platform-wide solution for this type of file inclusion. if you'd like access to the short term next-specific plugin solution, feel free to email me [email protected] or dm on twitter @levlinds. otherwise, i'll leave this issue open until the longer term solution rolls out! <3
Reaching out to you on twitter for this magical temporary solution!
from next-runtime.
Thanks for the good news @lindsaylevine! Is this also intended to work with Server-side Rendered pages using getServerSideProps
? I ran a quick test but couldn't get it to work. If there are any examples available of this in use with SSR then that would be amazing to see (understand if that's not possible though!)
from next-runtime.
@coreyaus hey! yes it should work with getServerSideProps! mind sharing how you're reading these files in your page, where the files are in your project, and how you configured your netlify.toml? that'd help me figure out why it's not working for you! :)
from next-runtime.
@lindsaylevine I tried, 3.4.0 included_files
for reading .json
but unfortunately it does not work when in preview mode. I get /var/task/public/json/xyz.json ...
not found. (also tried reading from the path /var/task/json/xyz.json
without any luck). Local
netlify build` in both production and preview work just fine. :(
My workaround has been using fetch
to get these json files but I'd love to remove this logic in the hope that it would speed up our preview mode experience.
How is process.cwd
support coming along?
Do you have any idea when this will be generally available and where would be the best way to find out when it's ready? - I ask specially as this seems as "small" feature in netlify's nextjs support scope
thanks in advance!
from next-runtime.
@juanpprieto the fix that @lindsaylevine mentioned has now been rolled out. Can you please check if it works for you? Thanks!
from next-runtime.
Thanks @eduardoboucas.
Unfortunately, I haven't had any luck. I just tried this on "@netlify/plugin-nextjs": "3.4.2", but I'm either not accessing .jsons correctly or it's not working.
netlify.toml
[functions]
node_bundler = "esbuild"
included_files = ["json/collections/*.json"]
.json & getStaticProps
My .jsons live on /public/collections/*.json
. I'm then trying to read the .json on the collection's getStaticProps with something like (pseudo-code):
const collection = await fs.readFile(path.join(process.cwd(), `/json/collections/${params.handle}.json`), 'utf8')
ENOENT: no such file or directory, open '/var/task/json/collections/tires.json'
Console logging process.cwd()
still returns /var/task
, so I get nothing back
Notes
I inspected the /out_functions/next_collections_handle.js
and I don't see any references to these json files.
Any idea what I may be doing wrong?
P.S reading from the filesystem works fine on every context except (isPreview === true && NODE_ENV === 'production')
from next-runtime.
Related Issues (20)
- [Bug]: Plugin "@netlify/plugin-nextjs" internal error: ENOENT: no such file or directory with runtime 5.0.0-beta.7 HOT 9
- [Bug]: Issue with data revalidation after new build HOT 3
- [Bug]: NETLIFY_NEXT_PLUGIN_SKIP no longer works in v5 HOT 7
- Nonce not automatically set in script tags when using CSP
- Middleware on sites with i18n cannot rewrite to static files
- Middleware should not add trailing slashes to non-data requests in static dir
- Pages router middleware should return 302 status for redirected data requests
- Server error pages return encoded data without `content-encoding` header if `accept-encoding` is `gzip`
- Fetch action prefetch cache test failing
- Appending `set-cookie` header in middleware leads to duplicate header
- Pages router data requests returning 404 when middleware is used
- Middleware does not match when using basePath and default locale
- Middleware matching is too broad when using i18n
- Does not correctly handle user middleware that redirects to path with canonical locale casing when app enables `skipMiddlewareUrlNormalize` and path contains locale slug with non-canonical casing
- 5.3.3 build error HOT 5
- cancel
- Local builds: `netlify build` followed by `netlify deploy` doesn't work with Next.js sites HOT 2
- 404 error in css and js files HOT 4
- Plugin "@netlify/plugin-nextjs" must not be specified twice in the app HOT 1
- SyntaxError - internal error during deploy within netlify/edge-functions HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from next-runtime.