Git Product home page Git Product logo

infrastructure-puppet's Introduction

jQuery — New Wave JavaScript

Meetings are currently held on the matrix.org platform.

Meeting minutes can be found at meetings.jquery.org.

Contribution Guides

In the spirit of open source software development, jQuery always encourages community code contribution. To help you get started and before you jump into writing code, be sure to read these important contribution guidelines thoroughly:

  1. Getting Involved
  2. Core Style Guide
  3. Writing Code for jQuery Projects

References to issues/PRs

GitHub issues/PRs are usually referenced via gh-NUMBER, where NUMBER is the numerical ID of the issue/PR. You can find such an issue/PR under https://github.com/jquery/jquery/issues/NUMBER.

jQuery has used a different bug tracker - based on Trac - in the past, available under bugs.jquery.com. It is being kept in read only mode so that referring to past discussions is possible. When jQuery source references one of those issues, it uses the pattern trac-NUMBER, where NUMBER is the numerical ID of the issue. You can find such an issue under https://bugs.jquery.com/ticket/NUMBER.

Environments in which to use jQuery

  • Browser support
  • jQuery also supports Node, browser extensions, and other non-browser environments.

What you need to build your own jQuery

To build jQuery, you need to have the latest Node.js/npm and git 1.7 or later. Earlier versions might work, but are not supported.

For Windows, you have to download and install git and Node.js.

macOS users should install Homebrew. Once Homebrew is installed, run brew install git to install git, and brew install node to install Node.js.

Linux/BSD users should use their appropriate package managers to install git and Node.js, or build from source if you swing that way. Easy-peasy.

How to build your own jQuery

First, clone the jQuery git repo.

Then, enter the jquery directory, install dependencies, and run the build script:

cd jquery
npm install
npm run build

The built version of jQuery will be placed in the dist/ directory, along with a minified copy and associated map file.

Build all jQuery release files

To build all variants of jQuery, run the following command:

npm run build:all

This will create all of the variants that jQuery includes in a release, including jquery.js, jquery.slim.js, jquery.module.js, and jquery.slim.module.js along their associated minified files and sourcemaps.

jquery.module.js and jquery.slim.module.js are ECMAScript modules that export jQuery and $ as named exports are placed in the dist-module/ directory rather than the dist/ directory.

Building a Custom jQuery

The build script can be used to create a custom version of jQuery that includes only the modules you need.

Any module may be excluded except for core. When excluding selector, it is not removed but replaced with a small wrapper around native querySelectorAll (see below for more information).

Build Script Help

To see the full list of available options for the build script, run the following:

npm run build -- --help

Modules

To exclude a module, pass its path relative to the src folder (without the .js extension) to the --exclude option. When using the --include option, the default includes are dropped and a build is created with only those modules.

Some example modules that can be excluded or included are:

  • ajax: All AJAX functionality: $.ajax(), $.get(), $.post(), $.ajaxSetup(), .load(), transports, and ajax event shorthands such as .ajaxStart().

  • ajax/xhr: The XMLHTTPRequest AJAX transport only.

  • ajax/script: The <script> AJAX transport only; used to retrieve scripts.

  • ajax/jsonp: The JSONP AJAX transport only; depends on the ajax/script transport.

  • css: The .css() method. Also removes all modules depending on css (including effects, dimensions, and offset).

  • css/showHide: Non-animated .show(), .hide() and .toggle(); can be excluded if you use classes or explicit .css() calls to set the display property. Also removes the effects module.

  • deprecated: Methods documented as deprecated but not yet removed.

  • dimensions: The .width() and .height() methods, including inner- and outer- variations.

  • effects: The .animate() method and its shorthands such as .slideUp() or .hide("slow").

  • event: The .on() and .off() methods and all event functionality.

  • event/trigger: The .trigger() and .triggerHandler() methods.

  • offset: The .offset(), .position(), .offsetParent(), .scrollLeft(), and .scrollTop() methods.

  • wrap: The .wrap(), .wrapAll(), .wrapInner(), and .unwrap() methods.

  • core/ready: Exclude the ready module if you place your scripts at the end of the body. Any ready callbacks bound with jQuery() will simply be called immediately. However, jQuery(document).ready() will not be a function and .on("ready", ...) or similar will not be triggered.

  • deferred: Exclude jQuery.Deferred. This also excludes all modules that rely on Deferred, including ajax, effects, and queue, but replaces core/ready with core/ready-no-deferred.

  • exports/global: Exclude the attachment of global jQuery variables ($ and jQuery) to the window.

  • exports/amd: Exclude the AMD definition.

  • selector: The full jQuery selector engine. When this module is excluded, it is replaced with a rudimentary selector engine based on the browser's querySelectorAll method that does not support jQuery selector extensions or enhanced semantics. See the selector-native.js file for details.

Note: Excluding the full selector module will also exclude all jQuery selector extensions (such as effects/animatedSelector and css/hiddenVisibleSelectors).

AMD name

You can set the module name for jQuery's AMD definition. By default, it is set to "jquery", which plays nicely with plugins and third-party libraries, but there may be cases where you'd like to change this. Pass it to the --amd parameter:

npm run build -- --amd="custom-name"

Or, to define anonymously, leave the name blank.

npm run build -- --amd
File name and directory

The default name for the built jQuery file is jquery.js; it is placed under the dist/ directory. It's possible to change the file name using --filename and the directory using --dir. --dir is relative to the project root.

npm run build -- --slim --filename="jquery.slim.js" --dir="/tmp"

This would create a slim version of jQuery and place it under tmp/jquery.slim.js.

ECMAScript Module (ESM) mode

By default, jQuery generates a regular script JavaScript file. You can also generate an ECMAScript module exporting jQuery as the default export using the --esm parameter:

npm run build -- --filename=jquery.module.js --esm
Factory mode

By default, jQuery depends on a global window. For environments that don't have one, you can generate a factory build that exposes a function accepting window as a parameter that you can provide externally (see README of the published package for usage instructions). You can generate such a factory using the --factory parameter:

npm run build -- --filename=jquery.factory.js --factory

This option can be mixed with others like --esm or --slim:

npm run build -- --filename=jquery.factory.slim.module.js --factory --esm --slim --dir="/dist-module"

Custom Build Examples

Create a custom build using npm run build, listing the modules to be excluded. Excluding a top-level module also excludes its corresponding directory of modules.

Exclude all ajax functionality:

npm run build -- --exclude=ajax

Excluding css removes modules depending on CSS: effects, offset, dimensions.

npm run build -- --exclude=css

Exclude a bunch of modules (-e is an alias for --exclude):

npm run build -- -e ajax/jsonp -e css -e deprecated -e dimensions -e effects -e offset -e wrap

There is a special alias to generate a build with the same configuration as the official jQuery Slim build:

npm run build -- --filename=jquery.slim.js --slim

Or, to create the slim build as an esm module:

npm run build -- --filename=jquery.slim.module.js --slim --esm

Non-official custom builds are not regularly tested. Use them at your own risk.

Running the Unit Tests

Make sure you have the necessary dependencies:

npm install

Start npm start to auto-build jQuery as you work:

npm start

Run the unit tests with a local server that supports PHP. Ensure that you run the site from the root directory, not the "test" directory. No database is required. Pre-configured php local servers are available for Windows and Mac. Here are some options:

Essential Git

As the source code is handled by the Git version control system, it's useful to know some features used.

Cleaning

If you want to purge your working directory back to the status of upstream, the following commands can be used (remember everything you've worked on is gone after these):

git reset --hard upstream/main
git clean -fdx

Rebasing

For feature/topic branches, you should always use the --rebase flag to git pull, or if you are usually handling many temporary "to be in a github pull request" branches, run the following to automate this:

git config branch.autosetuprebase local

(see man git-config for more information)

Handling merge conflicts

If you're getting merge conflicts when merging, instead of editing the conflicted files manually, you can use the feature git mergetool. Even though the default tool xxdiff looks awful/old, it's rather useful.

The following are some commands that can be used there:

  • Ctrl + Alt + M - automerge as much as possible
  • b - jump to next merge conflict
  • s - change the order of the conflicted lines
  • u - undo a merge
  • left mouse button - mark a block to be the winner
  • middle mouse button - mark a line to be the winner
  • Ctrl + S - save
  • Ctrl + Q - quit

QUnit Reference

Test methods

expect( numAssertions );
stop();
start();

Note: QUnit's eventual addition of an argument to stop/start is ignored in this test suite so that start and stop can be passed as callbacks without worrying about their parameters.

Test assertions

ok( value, [message] );
equal( actual, expected, [message] );
notEqual( actual, expected, [message] );
deepEqual( actual, expected, [message] );
notDeepEqual( actual, expected, [message] );
strictEqual( actual, expected, [message] );
notStrictEqual( actual, expected, [message] );
throws( block, [expected], [message] );

Test Suite Convenience Methods Reference (See test/data/testinit.js)

Returns an array of elements with the given IDs

q( ... );

Example:

q("main", "foo", "bar");

=> [ div#main, span#foo, input#bar ]

Asserts that a selection matches the given IDs

t( testName, selector, [ "array", "of", "ids" ] );

Example:

t("Check for something", "//[a]", ["foo", "bar"]);

Fires a native DOM event without going through jQuery

fireNative( node, eventType )

Example:

fireNative( jQuery("#elem")[0], "click" );

Add random number to url to stop caching

url( "some/url" );

Example:

url("index.html");

=> "data/index.html?10538358428943"


url("mock.php?foo=bar");

=> "data/mock.php?foo=bar&10538358345554"

Run tests in an iframe

Some tests may require a document other than the standard test fixture, and these can be run in a separate iframe. The actual test code and assertions remain in jQuery's main test files; only the minimal test fixture markup and setup code should be placed in the iframe file.

testIframe( testName, fileName,
  function testCallback(
      assert, jQuery, window, document,
	  [ additional args ] ) {
	...
  } );

This loads a page, constructing a url with fileName "./data/" + fileName. The iframed page determines when the callback occurs in the test by including the "/test/data/iframeTest.js" script and calling startIframeTest( [ additional args ] ) when appropriate. Often this will be after either document ready or window.onload fires.

The testCallback receives the QUnit assert object created by testIframe for this test, followed by the global jQuery, window, and document from the iframe. If the iframe code passes any arguments to startIframeTest, they follow the document argument.

Questions?

If you have any questions, please feel free to ask on the Developing jQuery Core forum or in #jquery on libera.

infrastructure-puppet's People

Contributors

atdt avatar krinkle avatar mgol avatar supertassu avatar timmywil avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

infrastructure-puppet's Issues

Upgrade from Debian 7 Wheezy (Puppet 3) to Debian 11 Bullseye (Puppet 7)

In order to get away from the very outdated Debian versions and such, we need to also get to a newer Puppet version.

We are currently using numerous Puppet 2 features that were deprecated in Puppet 3 and removed in Puppet 4. The main change that I think affects us is the change from "environment configs" to "environment directories".

Some relevant links:

Status quo: Puppet 3

The puppet server runs at puppet.ops.jquery.net (in legacy docs: puppet-master). The config for the server is at /etc/puppet/puppet.conf. There are two Git clones that we care about on this server:

  • /etc/puppet - This is a clone of jquery/infrastructure.git at branch puppet-master. This currently replaces the entire /etc/puppet directory.
  • /etc/puppet-stage – This is a directory we made up, containing another clone of jquery/infrastructure.git at branch puppet-stage.

In /etc/puppet/puppet.conf (the only place the Puppet server actually looks at) we have the following stuff:

[main]
# …
templatedir=$confdir/templates
manifest=/etc/puppet/manifests/site.pp

[stage]
manifest=/etc/puppet-stage/manifests/site.pp
modulepath=/etc/puppet-stage/modules
# …

[master]
# …

By default, with one of our droplets that runs a puppet agent asks for provisioning, it gets provisoned by the main config which points simply at the subdirectories within /etc/puppet. On staging hosts, we have another /etc/puppet/puppet.conf file that may contain environment = stage, which the agent passes on to the Puppet server, and so the Puppet server will consider that manifest and modulepath directory instead (in addition to compling it with $::environment = "stage").

Beyond this, the only other thing worth knowing is that we use jquery::postreceive instances (similar to for the content sites) to automatically update these git checkouts after commits to them. The actual applying of changes however is passive, based on puppet agents checking in with the server every 30 minutes (default Puppet agent behaviour).

Puppet 4

Under Puppet 4, things are a little bit different. There is no longer support for the templatedir, manifest, and modulepath parameters, and there is no longer support for per-environment configuration section overrides.

Instead, modules are read from a directory like /etc/puppet/code/environments/:environment/modules and manifests are read from a directory like /etc/puppet/code/environments/:environment/manifests. For example: /etc/puppet/code/environments/production/modules.

I think global templates are no longer supported, or at least not varying by environment. But that's okay, we only have one file in /templates and that'll either just not support staging or maybe we can even get rid of it (do we still use Zabbix?).

The new directory layout seems feasible, we just create two more clones and keep both for a little while.

Transition

I noticed just now that, apart from a few minor tweaks being needed for deprecated features, more generally it is not supported to connect Puppet 4 clients to a Puppet 3 server. However, the other way around is supported. So, the puppet master will have to go first, and that means a master switch, and setting up a new one of those first as well.

The good news is, a Puppet server is relatively easy to configure and gradually switch to...

Dedicated tickets:

Improve gilded-wordpress to tolerate newer server if semver compatible

Right now, the moment gilded-wordpress is updated on the server-side, clients in content repos like api.jquery.com are broken unless all 12 repos are updated at the same time.

Note that in practice, nothing breaks because:

  1. We ignore the lock file (#38).
  2. These aren't long-running services, so nothing actually "breaks". It just means you have to, at some point, update package.json in the content repos before the next time you want to deploy changes in content from that repo. The content repos are quite stable, so it's perfectly fine for them to be out of sync for a few weeks until someone actually wants to deploy from them and push a tag. Until then, there's no issue either way.

confirm jQuery CDN is accessible to Tor clients

We have received anecdotal reports from jQuery CDN users that StackPath/Highwinds may be blocking Tor clients.

It is not uncommon for CDN providers to offer WAF protection to avoid abuse, e.g. when serving a blog open to comments, or some other kind of dynamic service with an abuse vector. I'm guessing Highwinds has a blocklist of sorts that includes IPs of customers who happen to run Tor relays at home.

I was not able to find any kind of WAF or traffic filtering rules in Highwinds StrikeTracker, nor could I find anything about it in the Highwinds StrikeTracker support pages. However, the general StackPath support pages do mention their WAF service, and indeed that offers a preset for TOR exit nodes. I'm guessing a version of this rule is implicitly turned on for Highwinds, without any ability to turn off, or at least not in a way we can control ourselves.

https://support.stackpath.com/hc/en-us/articles/360001091666-Review-and-Allowlist-CDN-WAF-IP-Blocks

Depending on the timeline for switching from StackPath Highwinds to Fastly, it might not be worth escalating with StackPath. Instead, we can make sure that post-switch we can check and make sure Fastly does not block access to jQuery CDN from home IPs that use Tor.


Ref jquery/codeorigin.jquery.com#95, \cc @vincejv.

"PHP Warning: Attempt to read property on null" in themes/learn.jquery.com/sidebar.php

krinkle@wp-04:/var/log/nginx$ tail -f error.log 
PHP Warning:  Attempt to read property "post_parent" on null
in /srv/wordpress/jquery-wp-content/themes/learn.jquery.com/sidebar.php on line 35
PHP Warning:  Attempt to read property "ID" on null
in /srv/wordpress/jquery-wp-content/themes/learn.jquery.com/sidebar.php on line 15

PHP Warning:  Attempt to read property "post_parent" on null
in /srv/wordpress/jquery-wp-content/themes/learn.jquery.com/sidebar.php on line 35
PHP Warning:  Attempt to read property "ID" on null
in /srv/wordpress/jquery-wp-content/themes/learn.jquery.com/sidebar.php on line 35
krinkle@wp-04:/var/log/nginx$ ls -halF
1.1G error.log

This is reproducible via URLs such as https://learn.jquery.com/?s=findnothingnothingatall

TestSwarm: Auto-provision projects and project tokens (then: rotate passwords)

In the team meeting on 31 March 2023, we looked back on the swarm-02.ops server refresh (https://github.com/jquery/infrastructure/issues/444). The main thing that stood out is the lack of backup, which, while intentional, is also a bit of a weakness.

The most prominent issue was the runtoken used by testswarm-browserstack clients, which we've since then fixed by adding the capability in TestSwarm for the runtoken to be provisioned through a configuration file (jquery/testswarm@a7e7d6a, c089253).

The remaining issues are:

  • Auto-provisioning of the project accounts. We currently have to create these manually based on private documentation at https://github.com/jquery/infrastructure/wiki/TestSwarm#create-users. This includes very outdated passwords, which we should rotate and keep in our password vault instead.
  • Auto-provisioning of project tokens. These are currently dynamically generated as random values at install time, yet, we reference them statically from private Jenkins configuration files (jenkins: node-testswarm-config.json). Ideally, these too would be auto-provisioned via Puppet secrets and thus be able to to be kept in sync, or at least be declared and re-creatable on a fresh server in the same way so that server refresh and token refresh are seperable events.

Decom CLA droplets

The https://cla.js.foundation site is configured in Cloudflare, and has the jsf-cla-assistant droplet (IP 159.203.165.250, created 23 Oct 2016) as its backend. This droplet appears not managed by either Puppet or Ansible, and I'm unable to SSH into this myself. @brianwarner Do you have access to this one?

Screenshot of https://cla.js.foundation from before I powered off the droplet just now:
Screenshot

We also have the former cla.jquery.net site, backed by the cla-01.ops.jquery.net droplet in DigitalOcean (IP 104.131.146.50, created 5 Feb 2015), which I do have access to and is managed by Puppet. This uses the software at https://github.com/jquery/jquery-license, and was until recently also used dynamically by the https://contribute.jquery.org website through URLs like http://contribute.jquery.org/CLA/status/?owner=jquery&repo=jquery&sha=XYZ.

Next steps:

  • Determine whether the CLA data on cla-01.ops.jquery.net needs to be extracted and preserved somewhere. If yes, @Krinkle can help with this.
  • Determine who has access to the https://cla.js.foundation server.
  • Determine what's on the https://cla.js.foundation database.
  • If it needs to be extracted and preserved, do so.
  • Delete the cla-01.ops.jquery.net droplet, and remove CLA Puppet manifests.
  • Delete the jsf-cla-assistant droplet.
  • Set up redirect for cla.js.foundation, similar to what we do with contribute.jquery.org/CLA, docs.jquery.com view.jquery.com, etc already.

Upgrade from Debian 11 Bullseye to Debian 12 Bookworm

Main differences:

Debian 11 Bullseye hosts today:

  • wp-02.stage.ops.jquery.net
  • builder-04.stage.ops.jquery.net
  • puppet-03.ops.jquery.net
  • search-02.ops.jquery.net #36
  • codeorigin-02.stage.ops.jquery.net
  • codeorigin-02.ops.jquery.net
  • wpblogs-01.ops.jquery.net
  • gruntjs-02.stage.ops.jquery.net
  • gruntjs-02.ops.jquery.net
  • miscweb-01.ops.jquery.net
  • contentorigin-02.ops.jquery.net
  • swarm-02.ops.jquery.net

The following went straight from legacy Debian 7 to Debian 12 Bookworm, via #8, and were never on Debian Bullseye.

  • wp-*.ops
  • builder-*.ops
  • filestash-*.ops

Transition jQuery CDN from StackPath to Fastly

General

  • Document recent traffic profile. https://github.com/jquery/infrastructure-puppet/blob/be7518e07a/doc/cdn.md#latest-statistics
  • Document current CDN settings at StackPath (from Highwinds StrikeTracker). https://github.com/jquery/infrastructure-puppet/blob/be7518e07a/doc/cdn.md#highwinds-configuration
  • Create Fastly account, set up delegate access to 2+ admins with 2FA enabled.
  • TLS: Upload custom *.jquery.com certificate.
  • DNS: We prefer CNAME flattening to reduce lookups. Okay?
  • DNS We generally prefer 24h TTL to reduce lookups (shorter during switchover). Okay?
  • DNS: Figure out the correct entrypoint that satisfies out TLS and Networking preferences:
    • Dual stack IPv4 + IPv6.
    • HTTPS with HTTP2 and HTTP1.1
    • HTTP with HTTP1.1 (no redirects).
    • TLS 1.2+ configured such that it is compatible with at least IE9/Win7 for compat with current setup and customer expectations. Ref #21.
  • Service: Gzip enabled with strongest settings.
  • Service: Ignore URL query parameters for caching, to reduce origin load.
  • Service: Treat URLs as case-insensitive such that /jQuery-foo.js is able to match /jquery-foo.js.
  • Final confirmation that account is ready to handle 2.2 PB bandwidth per month with peaks of 30K req/s and 8.9Gbps (see traffic profile). E.g. no relevant limitations, quotas, or trial modes in place.

Testing

  • Compression don't poison the cache (either split, or shared and decompressed by edge).
  • Case insensitive URLs don't poison the cache.
  • Various desktop and mobile browsers on real devices.
  • Use curl to try every combination of -4, -6, --http1.1, --http2, --tls-max 1.2, --tls-max 1.3, http+https URLs (except http2 over HTTP) and confirm HTTP 200 OK (esp no redirect). Use --connect-to ::SOMETHING.global.fastly.net to test prior to deploying any DNS changes.

Deployment

Three services overall: code, content, releases.

  1. code: Switch low-traffic alias codeorigin.jquery.com for functional testing.
  2. content: Switch completely, including aliases.
  3. releases: Switch stage.releases.jquery.com for functional testing.
  4. releases: Switch releases.jquery.com. First significant exposure. This is aimed at developers during development, not in production, not in critical path.
  5. code: Update our high-traffic doc sites https://jquery.com and https://api.jquery.com to use codeorigin.jquery.com instead of code.jquery.com. This significantly increases exposure to learn of any connectivity issues that may be specific to uncommon browsers, geography/ISPs, firewalls.
  6. code: The big one Switch code.jquery.com.
  7. code: Switch our high-traffic doc sites back to using the "code.jquery.com" canonical name.

Examples of past issues:

Post-deployment

  • Update sponsorship message on the jQuery CDN homepage, and in the footer of content sites. Ref https://github.com/jquery/jquery-wp-content
  • Update technical docs to remove or update procedures and references to CDN providers.

puppet agent reports version as "" empty string

Regression caused by 864f584.

It seems as-is, we are mutually exclusive between octodiff working (and real puppet showing empty string as version) or octodiff failing on a git command (and real puppet showing commit message).

Enable WordPress automatic updates

Enable WordPress automatic updates so that we don't have to be constantly upgrading it by hand, with an option to disable those and roll back to a specific version via Puppet.

TypeError: str_contains() argument must be of type string, array given in wp-login.php

Various bots and crawlers are producing entries like the following in wp-05:/var/log/php8.2-fpm.log:

[15-Sep-2023 15:02:17] WARNING: [pool www] child 2355747 said into stderr:
  PHP Fatal error:  Uncaught TypeError: str_contains(): Argument #1 ($haystack) must be of type string, array given
  in /srv/wordpress/sites/api_jquery_com/wp-login.php:1365
  Stack trace:
  #0 /srv/wordpress/sites/api_jquery_com/wp-login.php(1365): str_contains()
  #1 {main}
  thrown in /srv/wordpress/sites/api_jquery_com/wp-login.php on line 1365

Seems to be an upstream issue where a $_GET or $_REQUEST key is checked for existence but not for type, thus prone to misuse when crafting query parameters in the array-form that PHP supports.

https://github.com/WordPress/wordpress-develop/blob/6.3.1/src/wp-login.php#L1267-L1365

Enable WordPress-level backups on docs sites

Even though those can be re-built in case something goes wrong, let's enable Tarsnap backups of the WordPress databases and content so that we have a faster way to recover from a failed WordPress upgrade or similar issue.

Set 'Cross-Origin-Resource-Policy: cross-origin' header on jQuery CDN responses

@mikewest wrote at jquery/codeorigin.jquery.com#57:

https://docs.google.com/document/d/1zDlfvfTJ_9e8Jdc8ehuV4zMEu9ySMCiTGMS9y0GU92k/edit#bookmark=id.kaco6v4zwnx2 is part of an explainer for the general approach browsers are taking.

[…] Digging through HTTP Archive, I see ~158k sites depending on a script resource of some sort from code.jquery.com.

@mikewest wrote at MaxCDN/bootstrapcdn#1495:

Yes, Cross-Origin-Resource-Policy: cross-origin is what you'd apply to resources that ought to be embeddable across the web.

Ensure output from node-notifier build failures is easy to retreive

When a webhook fires and rebuilds a content site like api.jquery.com, and it a command like npm install or grunt deploy failures, the output of those commands should be easy to retrieve from the builder host. For example, via /var/log/syslog, or via sudo journalctl -u notifier-server -f -n100.

The last time I tried this, however, the output was not readily available. It seems either the appropiate log levels are turned off, or perhaps the (sub?) process output isn't captured at all from the systemd unit perspective.

Some of the past findings from ~2021 are saved at https://github.com/jquery/infrastructure/wiki/Builder-host.

Switch wpblogs server to use jquery-wp-content instead of fork at blog.jquery.com-theme

blog.jquery.com-theme is effectively a fork of jquery-wp-content.

Proposed:

  • Update the fork with the latest "jquery" theme from jquery-wp-content.
  • Update jquery-wp-content with the latest blog theme, including missing theme for jqueryui and jquerymobile.
  • Switch wpblogs install to use jquery-wp-content instead.

@supertassu Are you aware of aspects of jquery-wp-content (specifically, the "next" branch) that would make it difficult to use for the blogs? E.g. things that perhaps the blog fork has stripped out that might pose issues?

The main thing I can think of is the multi-site aspects, but we've weeded those out on for the new wpdocs servers where we don't use multi-site WordPress, so I suspect it'd be fine, but curious what you think.

Renew star.jquery.com cert (expires 14 July 2023)

Previous renewal at https://github.com/jquery/infrastructure/issues/551, with previous testing methodology and results at https://github.com/jquery/infrastructure/issues/551.

Timeline:

For future reference, please note that the turnaround time was quick in part due to escalation by Benjamin Sternthal and in part because Christopher was already familiar with me and my public key from the year before. I would recommend if someone else requests these in the future, to pair the original request with your GPG public key, and make sure to confirm that you want to receive it on an email address matching your GPG key.

  • Tue 11 July 2023: Decrypted the .key file, and generated the .pem file as per the README instructions in /modules/jquery/files/cert/. And subsequently verify the file using the verify_certs.sh script before uploading anywhere else.
  • Tue 11 July 2023: Changed Cloudflare settings for one lower-traffic domain (https://learn.jquery.com) to disable proxying, so that we can expose the wp-01.ops.jquery.net droplet directly for that site, thus testing the new certificate. Confirm in a web browser that the used certificate is indeed the new one ("Valid not before" some recent date, "Valid not after" Next year).
  • Tue 11 July 2023: Invite people in #jquery_dev:gitter.im on Matrix to test against https://learn.jquery.com from their various devices and command-line clients.
  • Tue 11 July 2023: Upload the crt/key/ca-bundle files to Highwinds StrikeTracker without making it the default. Confirmed that Highwinds' own internal checks are all green.
Screenshot
  • Wed 12 July 2023: Wait at least 24h (preferably 48h) after the certificate's start date, to account for clients with broken clocks (as per README and referenced research paper by Google). The cert became valid July 11 00:00:00 UTC, so preferably live on or after July 13 00:00:00. On the other hand, in this case we're also very close to the expiry of 14 July 2023, which creates the inverse problem, so we're forced to make a compromise.
  • Wed 12 July 2023 16:00: After about 36 hours of the cert being valid, and still more than 24 hours before the old cert expires, I've toggled the new cert as the default in Highwinds configuration.

TestSwarm: Enable automatic database prune

In the team meeting on 31 Mar 2023 we decided that "Yes", we do want to enable automatic pruning instead of relying on ad-hoc pruning when the database "gets too big". @mgol okay'ed an initial retention period of 1 year. We can run it daily or weekly.

If we do find we have to run it ad-hoc earlier due to space issues, we can follow-up by tweaking the parameters in the cronjob, but we'll have it in-place at least.

Disable cron or move to CLI

I noticed in Fastly Observer for releases.jquery.com that there's a handful of "Pass" requests once every few minutes (i.e. not cache-miss requests, but uncachable requests, despite having a fallback TTL).I set up a temporary log bin, to look at what these are and they turn out to be POST requests, such as:

16:39
    /xmlrpc.php POST
    /wp-cron.php?doing_wp_cron=*** POST
16:50
    /xmlrpc.php POST
    /wp-cron.php?doing_wp_cron=*** POST
16:51
    /wp-cron.php?doing_wp_cron=*** POST
    /wp-cron.php?doing_wp_cron=*** POST

We can drop these at the edge, since the builder communicates directly with the origin anyway, and the default web-based cron can be turned off in WordPress. We can then turn it back on via a systemd time once a day or something.

https://developer.wordpress.org/plugins/cron/understanding-wp-cron-scheduling/
https://developer.wordpress.org/plugins/cron/hooking-wp-cron-into-the-system-task-scheduler/

Looking at wp_options where option_name='cron' shows sites typically have the following recurring tasks:

hourly wp_privacy_delete_old_export_files # unused in our headless setup
daily wp_scheduled_auto_draft_delete
twicedaily wp_https_detection
twicedaily wp_version_check
twicedaily wp_update_plugins
twicedaily wp_update_themes
daily recovery_mode_clean_expired_keys
weekly wp_delete_temp_updater_backups
weekly wp_site_health_scheduled_check

Document shell access creation and recommended ssh setup

@markelog wrote:

Idea - Keep known fingerprints to easily import to avoid having to trust first ssh connect.

@Krinkle wrote:

Maybe in public infra repo, and then document how to load it into ~/.ssh/known_hosts.d/ and how to enable it in ~/.ssh/config.

@supertassu wrote:

https://puppet-03.ops.jquery.net/known_hosts exists now. However it's not documented and it probably needs a more stable hostname if we want to encourage people to use it.
Or we could just have everyone trust the SSH CA: @cert-authority *.ops.jquery.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPt01ydjmlHiFKFD3ya6JcQtEPe0WbPj6JnGa/noy4mI jQuery SSH CA v1 (first line of above link)

Consolidate static.jquery.com and content.jquery.com

To simplify management, let's consolidate these two file servers since the paths don't overlap in conflicting ways, and a portion of their content is already duplicated/available on both.

This in turn let's us simplify the Fastly config to 1 site.

Re-enable GitHub backups

Follows-up from https://github.com/jquery/infrastructure/issues/9. We have a backup role in the Puppet manifest (github::backup), and it was used for a while on the jq02 host, but that host has since been decom'ed.

This is a task to review the script, fix it if needed, and then re-enable it in one of the host roles. It's pretty minor and does not need a dedicated host. It runs once a night and needs a bit bit of space. The builder host seems like a good candidate as it's the only host with any real space needs.

jQuery Blog fails converts some non-ASCII characters to question marks

The jQuery blog at https://blog.jquery.com/ doesn't support some non-ASCII characters, like the ones in my name: ł, ę. If you enter them anywhere, they get converted to question marks.

The issue looks related to the database as it's observed regardless of if you enter such a letter when authoring a blog post or in a plain text field when setting First/Last Name on the profile settings page.

The same issue doesn't exist in the UI (https://blog.jqueryui.com/) & Mobile (https://blog.jquerymobile.com/) blogs despite them being on the same WordPress version (5.8.1).

cc @timmywil @Krinkle

Make Trac read-only, possibly replace by a static dump

A while ago, we made jQuery Core Trac instance (https://bugs.jquery.com/) read-only, making the only still fully functional Trac instance to be the jQuery UI one (https://bugs.jqueryui.com/). With jQuery UI being maintained in a very limited way nowadays, it doesn't make sense to maintain a full separate bug tracker just for this project; other projects are using GitHub issues.

We want to enable GitHub issues for jQuery UI and make the UI Trac read-only. In the future, we can consider replacing Trac with a static dump of all its pages if that makes maintenance easier.

I'm not 100% sure what we did for Core but I think we mostly blocked account registration and removed all existing accounts as with no accounts the site is essentially read-only. @rjollos can you help with doing the same for UI? I'm not sure if it was you or someone else involved with the changes for the Core Trac.

Migrate search autocomplete from Algolia to Typesense

Background as documented previously:

As of 2021, we're exploring an open-source solution that we can support within the free software ecosystem. In doing so we will increase security and availability (by reducing client-side dependence on third-party domains), and lower our privacy budget.

We first evaluated Meilisearch and experienced some suboptimal aspects. These included: difficult upgrades (not yet committing to forward compatibility or automatic in-place upgrades), opt-out telemetry instead of opt-in, no official Debian packages, non-trivial interactive setup, missing support for querying multiple indexes (e.g. qunitjs.com and api.qunitjs.com), and a not yet clear future in terms of business model (Meilisearch Cloud was not yet in the picture, and the backend is not GPL licensed).

In mid-2022, the experiment transitioned to focus on Typesense instead.


Since April 2023 we have an instance of Typesense running in the new infra, provisioned through this repostory (558de96). I also developed a 2kB minimalistic HTML-first client and user interface for it at https://github.com/jquery/typesense-minibar and integrated it with our Jekyll theme at https://github.com/qunitjs/jekyll-theme-amethyst/. This has been live on https://qunitjs.com/ for the past few months.

Next, we need to migrate the remaining doc sites which are still using the (now stale and deprecated) Algolia DocSearch indexes:

Archive plugins.jquery.com, replace with static site

Follow up from #6, in which we moved all ~20 doc sites to new infrastructure as simple standalone WordPress sites. There is one site we haven't migrated yet: https://plugins.jquery.com/.

This looks like a fairly stateful site, and also lacks a staging site. It includes several custom build steps that we haven't ported over yet, and I'm actually not sure that it would re-create the same site even if we do run it from scratch since the underlying sources may have dissappeared or significantly changed.

We could try to copy and upgrade the existing database as-is, but maybe we want to use this chance to turn it into a static site (like #10). Possibly a bit simpler and slimmed down only an index listing with a page for each plugin (URL-compatible) showing the plugin meta data (description, author, links to website/docs/bugs), so that it's easy for people to find what this points to and where to find source code, updates, contact persons, or forks going forward.

Remaining items

  • Search index improvements
  • Update https://learn.jquery.com/plugins/ with npm authoring note
  • Deploy plugins site to desired platform
  • Remove existing plugins site droplet
  • Transfer plugins repo to the jQuery org

Upgrade WordPress hosts to PHP 8

The wp hosts are currently on PHP 5. More specifically, PHP 5.4, which has been EOL since 2014.

A safer option might be PHP 5.6, which has support for another year. On the other hand, there have been very few breaking changes and WordPress themselves currently recommend PHP 7.2. Seems worth trying to upgrade there in one step.

Quote from https://github.com/jquery/infrastructure/issues/312:

As part of the infra refresh we will (for now) continue to use WordPress for the foreseeable future. (ref #8, jquery/infrastructure#449, and subtask for WordPress: #6)

Notable changes:

To do:

  • jquery-blogs
    • blog.jquery.com
    • blog.jqueryui.com
    • blog.jquerymobile.com
  • wp-01
    • learn.jquery.com
    • api.jquery.com
    • jquery.com
    • plugins.jquery.com Being archived at #29
  • wp-02
    • *.jquery.org
    • *.jqueryui.com
    • *.jquerymobile.com
  • wp-03

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.