Dynamic Rendering is, as the name implies, about rendering a page based on a dynamic condition. What we want to do is solve the very common problem of modern Progressive Web Apps not working very well with some older crawlers that are unfortunately still in use across the web, such as on those used by Facebook and LinkedIn. Our condition is therefore ”if a crawler sees the page, give it prerendered markup”.
For this to work, we need to intercept the browser’s request BEFORE it actually hits the server to see whether the visitor is a bot or not. This repo assumes Amazon Web Services (AWS) for the entire solution. For the edge function capability, this responsibility could probably be doled out to Cloudflare Workers or similar as well.
The individual services used will be:
- S3 bucket for hosting a static page; I’ve included a very basic routed React application in the
public
folder if you need something to experiment with - CloudFront for adding edge locations and a CDN for the S3-hosted site
- Serverless Lambda@Edge functions to intercept the browser request
- Serverless ”regular” Lambda function to run a lightweight instance of Google’s headless browser, Puppeteer
All of this magic can be fairly tedious to work with from scratch, especially if you are not accustomed to AWS. Combined with the slow deployment times, it’s not really the most user-friendly mini-project I’ve done. I hope I spare some of you out there a bit of wasted effort!
The below helps with the overall steps for manually setting everything up. You should be able to set up with Terraform or similar as well. Lambda@Edge functions through Serverless has proven hard, even with a fairly well-documented package out, so unfortunately you will have to deploy them manually through the web frontend.
Curl or GET https://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender?url=http://d13x2tqlfdnb32.cloudfront.net/thatview
.
- You will need an AWS account
- Highly likely that you must have the AWS SDK installed
- Highly likely that you must be logged in through the terminal/environment
- Optional: A REST client like Insomnia if you don't really love curl'ing in your terminal
Use whatever you want!
A basic, routed React application is available in the s3
folder if you need something to experiment with. The routes are /thisview
and /thatview
.
- Have an AWS account and be logged in through the terminal/environment
- Install the dependencies in the repo with
yarn
ornpm install
- Run
sls deploy
to deploy the package with Serverless - Upon successful push, you will receive an endpoint URL, such as
https://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender?url=http://d13x2tqlfdnb32.cloudfront.net/thatview
- Test running a GET request to the prerenderer, like so:
https://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender?url=http://d13x2tqlfdnb32.cloudfront.net/thatview
- Copy the endpoint URL, and paste it into the
BASE_URL_RENDERER
constant infunctions/edgePrerenderResponse.js
(should be line 26)
- Go to https://s3.console.aws.amazon.com/s3/
- Deactivate anything to do with blocking access (for now, switch on things later as needed and what works)
- Turn on Static website hosting, point index and errors document fields to
index.html
- Before closing the panel, note the endpoint URL
- In the Permissions tab, add a Bucket Policy (located in
snippets/bucket-policy.json
) - In the Permissions tab, add a CORS configuration (located in
snippets/cors.xml
)
- Go to https://console.aws.amazon.com/cloudfront/
- Click Create new Distribution
- Make sure that Origin Domain Name is the endpoint URL from S3 (the one seen in ”Static website hosting” ex. http://prerender-demo.s3-website.eu-north-1.amazonaws.com)
- Set Allowed HTTP Methods to "GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE"
- Set cache TTL items to 0 when initially testing this (you probably want caching later, though!)
- Set Compress Objects Automatically to "Yes"
- Note the section named Lambda Function Associations, since this is where you will manually specify which functions are activated when requesting the CloudFront distribution
- Finish, then go to the Error Page tab
- Set Error caching time to 0
- Set Customize Error Response to "Yes"
- Set Response Page Path to
/index.html
- Set HTTP Response Code to 200
- The format is identical between all three error codes, except for the HTTP Error Code (400, 403, 404)
This is going to deploy for 10-15 minutes or so, so continue with the rest below.
These lambdas will be copy-pasted into the web code editor as support via Serverless is kind of wonky.
- Go to https://console.aws.amazon.com/lambda/
- Click Create new function
- Select Author from scratch
- Give it a good name; Node version should be 8.10
- In Execution Role, choose Create a new role with basic Lambda permissions
- For each function, copy-paste the supplied lambda code into the code editor
- Under Add triggers click CloudFront and then Deploy to Lambda@Edge
- You should be able to see the distribution you created (if not, you will have to wait for it to finish deploying)
- Cache behavior should be "*"
- The CloudFront event should correspond to whatever the supplied code is named (origin request, viewer response, etc.)
- Deploy!
- If you cannot deploy, you should do the next section in a new tab (you need to do it anyway)
- Repeat for all 4 functions
- Go to https://console.aws.amazon.com/iam/
- Go to Roles
- You should have roles created for each of the edge functions
I am overreaching a bit on what to create, but at least the following works for me.
- Under Permissions, click Attach policies
- Filter for
AWSLambdaBasicExecutionRole
and attach it - When it's added, click Add inline policy, and go to the JSON tab; paste in the contents of
snippets/log-policy.json
, save it - When it's added, go to the Trust relationships tab and click Edit trust relationship; paste in the contents of
snippets/trust-policy.json
, save it
- Go to https://console.aws.amazon.com/lambda/
- For each Lambda, note the ARN in the right corner (something like arn:aws:lambda:us-east-1:487572315234:function:PrerenderRequest:12)
- You must have the last part with
:12
or whatever the number might be - this corresponds to the version number, if you don't see it click the Qualifiers/Version box and switch to the version with the highest number - When you have collected all the four ARNs, go to https://console.aws.amazon.com/cloudfront/
- Go to Distribution settings and then to Behaviors
- If you don't have a behavior, create a new one, otherwise edit the previous
- Make sure to whitelist the following headers: Access-Control-Request-Headers, Access-Control-Request-Method, Origin, x-prerender-uri, x-resolved-user-agent, x-should-prerender
- At the bottom, in Lambda Function Associations open one event each for viewer request, origin request, origin response, viewer response
- In their respective Lambda Function ARN, paste in their corresponding ARN value including the
:12
(or whatever number) bit - Save
- Once again, everything will redeploy so have the CloudFront main view open to see the progress
- Facebook Sharing Debugger: https://developers.facebook.com/tools/debug/sharing/
- Google Mobile-Friendly Test: https://search.google.com/test/mobile-friendly
- LinkedIn: https://wwww.linkedin.com (will only show page title)
Check the default Cloudwatch logs at console.aws.amazon.com/cloudwatch/.
Edge functions will pop up in whatever region gets the request. This may not always be the most obvious region, but begin in the geographically closest region and start from there.
When you test Facebook or similar, the same goes for those: These are going to be run from one of the North American regions.
Many steps, but that should about do it to add dynamic rendering!