embroider-build / content-tag Goto Github PK

We need to make sure we've got SWC's typescript support enabled, so that we're able to preprocess GTS to TS just as well as GJS to JS.

But also, it would probably be nice to offer GTS to JS mode as well, since it's probably not hard to enable. This would potentially allow simpler dev setups since you could avoid adding a second tool to process the TS to JS.

stack trace:

one/Table.gts:10:7: 10:14
10:14:48 AM [vite] Internal server error: Parse Error at /Users/lifeart/Repos/glimmer-next/src/components/pages/page-one/Table.gts:10:7: 10:14
  Plugin: glimmer-next
  File: /Users/lifeart/Repos/glimmer-next/src/components/pages/page-one/Table.gts
      at module.exports.__wbg_Error_58015c683709e145 (/Users/lifeart/Repos/glimmer-next/node_modules/.pnpm/[email protected]/node_modules/content-tag/pkg/node/content_tag.cjs:294:17)
      at wasm://wasm/004e9cce:wasm-function[179]:0x6063b
      at wasm://wasm/004e9cce:wasm-function[396]:0x8dcc6
      at Preprocessor.process (/Users/lifeart/Repos/glimmer-next/node_modules/.pnpm/[email protected]/node_modules/content-tag/pkg/node/content_tag.cjs:238:18)
      at TransformContext.transform (file:///Users/lifeart/Repos/glimmer-next/vite.config.mts.timestamp-1705043533982-c0ee68bce3b52.mjs:977:32)
      at Object.transform (file:///Users/lifeart/Repos/glimmer-next/node_modules/.pnpm/[email protected][email protected]/node_modules/vite/dist/node/chunks/dep-uAHLeuC6.js:63685:62)
      at loadAndTransform (file:///Users/lifeart/Repos/glimmer-next/node_modules/.pnpm/[email protected][email protected]/node_modules/vite/dist/node/chunks/dep-uAHLeuC6.js:49384:51)
      at async viteTransformMiddleware (file:///Users/lifeart/Repos/glimmer-next/node_modules/.pnpm/[email protected][email protected]/node_modules/vite/dist/node/chunks/dep-uAHLeuC6.js:59005:32)

code sample:

export const Table = <template>
  <div></div>
</template>;

if (import.meta.hot) {
  import.meta.hot.accept((newModule) => {
    console.log('updated: count is now ', Object.keys(newModule));
    if (newModule) {
      console.log('updated: count is now ', newModule
      console.log('updated: count is now ', newModule.count);
    }
  });
}

right now the type is any. A better type might be something like:

interface ParseResult {
  type: "expression" | "class-member",
  tagName: "template",
  contents: string,
  range: {
    start: number,
    end: number,
  },
  contentRange: {
    start: number,
    end: number,
  },
  startRange: {
    end: number,
    start: number,
  },
  endRange: {
    start: number,
    end: number,
  },
}

// 
let result: ParseResult[] = p.parse(...);

This comes out of: embroider-build/addon-blueprint#195

Repro:

import { Preprocessor } from 'content-tag';
const p = new Preprocessor();


const gjs2 = `
import Component from '@glimmer/component';
import type { WithBoundArgs } from '@glint/template';
import { hash } from '@ember/helper';

import Header from './header.gts';
import Content from './content.gts';

export interface AccordionItemSignature {
  Element: HTMLDivElement;
  Blocks: {
    default: [
      {
        Header: WithBoundArgs<
          typeof Header,
          'value' | 'isExpanded' | 'toggle' | 'disabled'
        >;
        Content: WithBoundArgs<typeof Content, 'value' | 'isExpanded'>;
      },
    ];
  };
}

export default class AccordionItem extends Component<AccordionItemSignature> {
  /* ... */
  <template>
    ...
  </template>
}
`;

console.log(p.process(gjs2));

results in:

import { template } from "@ember/template-compiler";
import Component from '@glimmer/component';
import { WithBoundArgs } from '@glint/template';
import { hash } from '@ember/helper';
import Header from './header.gts';
import Content from './content.gts';
export interface AccordionItemSignature {
    Element: HTMLDivElement;
    Blocks: {
        default: [{
                Header: WithBoundArgs<typeof Header, 'value' | 'isExpanded' | 'toggle' | 'disabled'>;
                Content: WithBoundArgs<typeof Content, 'value' | 'isExpanded'>;
            }];
    };
}
export default class AccordionItem extends Component<AccordionItemSignature> {
    /* ... */ static{
        template(`
    ...
  `, {
            component: this,
            eval () {
                return eval(arguments[0]);
            }
        });
    }
}

not that the type import on @glint/template was stripped.

during the babel phase of the v2-addon build, this causes just WithBoundArgs to be removed, leaving a side-effecting import '@glint/template' -- because it's not safe to remove the entirety of value imports (which is why we want to require type annotations on type-imports.

gitKrystan/prettier-plugin-ember-template-tag#162

const num:   number = 1
(oops) => {}

import Component from '@glimmer/component'

/** It's a component */
class MyComponent
  extends Component {
    get whatever() {}

        <template>


    <h1>   Class top level template. Class top level template. Class top level template. Class top level template. Class top level template. </h1>
  </template>
  (oops) => {}
}

import type { TemplateOnlyComponent } from '@ember/component/template-only'

export interface Signature {
  Element: HTMLElement,
  Args: {


  }
  Yields: []
}


export const Exported:    TemplateOnlyComponent<Signature> = <template>       Exported variable template. Exported variable template.  Exported variable template.  Exported variable template.  Exported variable template. Exported variable template. Exported variable template. </template>
(oops) => {}

In handlebars, escapes like \n or \u1234 don't have any special meaning. If you use them in an .hbs file, they render as written. But if you convert that .hbs file to a template-tag, they are inserted directly into a Javascript string literal (or template literal) where they get interpreted, changing the rendered output.

This is not a bug unique to content-tag, the earlier ember-template-imports implementation also suffers this.

Our WASM bindings only offer a string-to-string function right now. But it would be nice to offer one that accepts a filename instead, so that the file reading happens on the rust side and avoids crossing the FFI boundary.

A content tag in module scope by itself it supposed to get an implied default export. We haven't implemented this yet.

When accepting a string input from javascript, we currently go &str to String to StringInput. Probably there is a more efficient way.

I would like to request for content-tag to also return source maps for the files it parses, so that we can map the parsed .js/.ts file back to the original .gjs/.gts file.

Example where this can be useful, when we are building a v2 addon with the standard v2 blueprint with rollup and let's say we want to use babel-plugin-istanbul to instrument our code, we would need the original source maps so that istanbul would know exactly how to map it's coverage to the original .gjs/.gts source files.

Use case (for me, anyway):

• https://limber.glimdown.com/
• https://tutorial.glimdown.com/
• https://ember-primitives.pages.dev/

This would fix an issue with Safari, where my copied ember-template-imports code's regex is not compatible with Safari.

We'll need to make sure no "extra stuff" (plugins, etc) are included in the build as browsers are more sensitive to extraneous bytes (on the embroider call today, Ed mentioned that he thinks this is already happening 🎉 ).

In order to get the index / replacement / splice management out of eslint-plugin-ember or other tooling, additional APIs may be needed -- or a unified AST may need to be provided -- or more tooling so folks don't have to manage the template removing / insertion again.

eslint-plugin-ember

• runs @glimmer/syntax over the template and re-stitches together AST nodes
  • enables type-aware linting
  • enables one lint tool (eslint) instead of two (template-lint)
    • enables fixers to be able to be implemented with the whole context of a module

prettier-plugin-ember-template-tag & ember-template-lint

• current strategy:
  • extract templates
  • format JS
  • format templates with offsets
  • place templates back in placeholder spots

Motivation

• index splicing is really easy to get wrong in javascript.
• any <template> manipulation really belongs with the tool that already is supporting content-tag, because it can more easily assure that parsing and serialization is correct.

Ideas:

format

makes the prettier plugin implementation easier.
the intermediary format is never seen.

import { format } from 'content-tag';

let outputGjsString = format(gjsString, (contentTagContent, positionInfo /* for reporting */) => {
  // no-op
  return contentTagContent;

  // or, 
  return prettier(contentTagContent, { parser: 'glimmer'});
});

parseAst + serializeAst

This would produce a JSON AST for the whole JS + <template> contents, and would require @glimmer/syntax for now, but I've volunteered myself to learn Rust and have a go at a rust-based glimmer template parser.

import { parseAst, serializeAst } from 'content-tag';
import { parse /* or whatever this was called */ } from '@glimmer/syntax';

// no <template> or intermediary format seen in here
let ast = parseAst(gjsString, { glimmerParse: parse });

let outputGjs = serializeAst(ast);

cc @patricklx @wagenet, @gitKrystan

version: content-tag 1.1.2

In investigating the root cause of gitKrystan/prettier-plugin-ember-template-tag#191 I discovered that content-tag is returning incorrect ranges when templates include multi-byte characters, such as emoji.

Reproduction:

For a 4-byte character:

import { Preprocessor } from 'content-tag';

const code = `import Component from '@glimmer/component';

class PooComponent extends Component {
  <template>💩</template>
}
`;

const p = new Preprocessor();
const templateNodes = p.parse(code); // Array of length 1

templateNodes[0].type
// 'class-member'

templateNodes[0].contents;
// '💩'

templateNodes[0].range
// {start: 86, end: 111}

code.slice(templateNodes[0].range.start, templateNodes[0].range.end)
// '<template>💩</template>
// }'

code.slice(templateNodes[0].endRange.start, templateNodes[0].endRange.end)
// 'template>
// }'

code.slice(templateNode.contentRange.start, templateNode.contentRange.end)
// '💩</'

Note that the range has gobbled up the following character(s).

Similarly, for a two-byte character:

import { Preprocessor } from 'content-tag';

const code = `import Component from '@glimmer/component';

class PoundComponent extends Component {
  <template>£</template>
}
`;

const p = new Preprocessor();
const templateNodes = p.parse(code); // Array of length 1

// code.slice(templateNode.contentRange.start, templateNode.contentRange.end)
'£<'

Interestingly, it gobbles fewer characters this time.

In the expression position, the issue is less noticeable, but still there:

import { Preprocessor } from 'content-tag';

const code = `<template>💩</template>
`;

const p = new Preprocessor();
const templateNodes = p.parse(code); // Array of length 1

templateNodes[0].type
// 'expression'

templateNodes[0].contents;
// '💩'

templateNodes[0].range
// {start: 0, end: 25}

code.slice(templateNodes[0].range.start, templateNodes[0].range.end)
// '<template>💩</template>
// '

code.slice(templateNode.contentRange.start, templateNode.contentRange.end)
// '💩</'

From our remaining TODOs:

When you run the example program in this repo with cargo run some-file-with-errors.gjs, you get nice terminal output explaining any syntax errors. But when you do the same thing through the WASM library bindings, we still provide a placeholder error that says "something went wrong".

Look at this file and port over the tests.

• copy tests over from ember-template-tag (a js implementation)
  • parseTemplates
    • parseTemplates does not exist in content-tag today
    • return shape in JS should be the same when implemented in content-tag
      • here is an exapmle of how the information is used
• implement the code in content-tag that makes the above implemented tests pass

inspo: #39 (comment)

Blocks:

• ember-template-lint/ember-template-lint#3021
• typed-ember/glint#655
• ember-tooling/ember-eslint-parser#56
• maybe others? (css experimentation, etc?)

I am reasonably confident that the span interning system in swc is going to hold onto an ever-growing amount of data if you continuously do rebuilds using the same Preprocess instance.

But every example in swc itself that I can find also does things that way. swc holds the source maps in a static lifetime and calls new_source_file again and again on it, which is equivalent to what we're doing.

It would be good for someone to exercise this on a bigger build and watch the memory consumption as soon as we have it wired into a useful system.

const { Preprocessor } = require('content-tag');

const p = new Preprocessor();

const res = p.process(`
f = function(this: Context, ...args) {
    function t(this: Context, ...args) {};
    <template></template>
}`);

console.log(res);

results to

import { template } from "@ember/template-compiler";
f = function(this: Context, ...args) {
   function t(this1: Context, ...args1) {}
   ;
   template(``, {
       eval () {
           return eval(arguments[0]);
       }
   });
};

removing the template will make it work correctly

Not only because it's nice to use typescript for our own tests, but because we want to ensure that the JS API typechecks correctly for typescript users.

wasm-pack is already generating d.ts files for us so in theory it already works.