itgalaxy / generate-robotstxt Goto Github PK
View Code? Open in Web Editor NEWGenerator robots.txt for node js
License: MIT License
Generator robots.txt for node js
License: MIT License
Hello,
I'm picking up this list of user agents at https://github.com/monperrus/crawler-user-agents/.
They have an entry containing "patterns" that could resolve into or more entries.
Does this library supports or could support such regexes?
For example
"pattern": "[wW]get",
yields these entries in robots.xt
wget
Wget
Thanks
Great work, thank you!
I wonder if there is any way to add a comments to the file. My use-case is to add a decoded version of the path I added for better readability.
Now: (ony encoded)
Disallow: */%D0%B7%D0%B0%D0%BF%D0%B8%D1%82%D0%B2%D0%B0%D0%BD%D0%B5$
Future: (+ decoded as comment)
...
Disallow: */%D0%B7%D0%B0%D0%BF%D0%B8%D1%82%D0%B2%D0%B0%D0%BD%D0%B5$
# Disallow: */запитване$
...
The following configuration:
{
"userAgent": "*",
"disallow": ""
}
ends up in the following robots.txt :
User-agent: *
instead of
User-agent: *
Disallow:
Branch | Build failing 🚨 |
---|---|
Dependency | remark-preset-lint-itgalaxy |
Current Version | 5.0.1 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As remark-preset-lint-itgalaxy is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.
I recommend you give this issue a high priority. I’m sure you can resolve this 💪
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot 🌴
Users/jay/Downloads/delete-express/app.js:37
}, callback)
^
ReferenceError: callback is not defined
Hi, I can’t understand the CLI part on the readme. How can I generate the robots.txt on plain html project? To deploy on netlify.
Branch | Build failing 🚨 |
---|---|
Dependency | ava |
Current Version | 0.18.0 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As ava is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.
I recommend you give this issue a high priority. I’m sure you can resolve this 💪
Quick bug fix: Remove t.is
and t.not
from enhanced assertion patterns to provide correct output on assertion failure.
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot 🌴
It takes as long Yandex permit rules are not more than 500 characters.
https://yandex.ru/support/webmaster/controlling-robot/robots-txt.xml?lang=ru#clean-param
Branch | Build failing 🚨 |
---|---|
Dependency | prettier |
Current Version | 1.4.4 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As prettier is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.
I recommend you give this issue a high priority. I’m sure you can resolve this 💪
This is the release I've been waiting for a very long time: one that has only minimal changes to JavaScript!
For the past 6 months, we kept doing changes to various aspects of printing JavaScript, with the hope that one day we would get to a stable place. No automated tool is going to print perfect code for all the possible edge cases. The goal is to find a good place where when people report code that is printed in a funny way, we can't make it better without making other pieces of code look worse, introduce behavior that's very hard to understand for humans and doesn't introduce some disproportionate complexity to the codebase.
We're not 100% there yet, but we're closer than ever!
Now that JavaScript needs for support is trending down, it's an opportunity to support other languages that front-end developers are working on and want formatted. We've introduced TypeScript and CSS in the last release and are doing a batch of fixes for them in this release. We're also adding support for new languages: GraphQL queries, embedding CSS-in-JS and JSON are now available in prettier!
Prettier is not only a useful tool but it's also a really cool piece of technology. @karl spent a bunch of time improving JSX support and in the process implemented a new primitive to prettier: fill
. He wrote a very interesting blog post Adding a new layout strategy to Prettier that I highly recommend reading if you're interested in how prettier is working behind the scenes.
Thanks to @stubailo, @jnwng, @tgriesser and @azz, prettier now supports printing GraphQL queries!
It works for .graphql
files and within JavaScipt templates that start with graphql
, graphql.experimental
and gql
in order to work with Relay and Apollo.
ReactDOM.render( <QueryRenderer query={graphql` query appQuery { viewer { ...TodoApp_viewer } } `} // ... />, mountNode );
Note that it only supports the open source syntax of GraphQL, therefore doesn't work with Relay Classic, it only works with Relay Modern.
If you are using styled-components or styled-jsx, prettier will now reformat the CSS inside of your template expressions.
const EqualDivider = styled.div` margin: 0.5rem; padding: 1rem; background: papayawhip; > * { flex: 1; &:not(:first-child) { ${props => (props.vertical ? "margin-top" : "margin-left")}: 1rem; } } `;
This was pretty straightforward to implement but nonetheless very useful. Thanks to @josephfrazier for doing it :)
{ "name": "prettier", "version": "1.5.0", "description": "Prettier is an opinionated JavaScript formatter", "bin": { "prettier": "./bin/prettier.js" } }
I'm really excited because we only put a few days to build the initial CSS support and it has worked surprisingly well. This release brings a handful of important improvements to CSS but nothing that required big changes.
The biggest unknown when printing CSS was how to deal with multiple selectors. The initial approach we took was to use the 80 columns rule where we would only split if it was bigger than that. Many people reported that they were using another strategy for this: always break after a ,
. It turns out that many popular codebases are using this approach and it feels good as you can see the structure of the selectors when layed out on-top of each others.
// Before .clusterPlannerDialog input[type="text"], .clusterPlannerDialog .uiTypeahead {// After
.clusterPlannerDialog input[type="text"],
.clusterPlannerDialog .uiTypeahead {
The concept of code formatting has blurry boundaries. The core aspect of it is around whitespaces but some things like single vs double quotes and semi-colons are usually bundled with it. With prettier on JavaScript, we also lightly reformat strings by removing extra \
and normalize numbers. For CSS, we need to do a similar interpretation of where the boundary ends. For colors, we decided to turn all the letters into lowercase and stop there. Turning rgb() into hex or 6 hex into 3 hex is out of scope.
// Before .foo { color: #AAA; -o-color: #fabcd3; -ms-color: #AABBCC; }// After
.foo {
color: #aa;
-o-color: #fabcd3;
-ms-color: #aabbcc;
}
The new fill primitive turned out to be very useful for CSS. For long values, instead of breaking and putting a \n
before every element, we can instead only put a \n
when it goes over the limit. It leads to much better looking code.
// Before border-left: 1px solid mix($warningBackgroundColors, $warningBorderColors, 50%);// After
border-left: 1px solid
mix($warningBackgroundColors, $warningBorderColors, 50%);
This is another small fix in the journey of properly supporting a new language. We now encode the ability to break on long @media
rules.
// Before @media all and (-webkit-min-device-pixel-ratio: 1.5), all and (-o-min-device-pixel-ratio: 3/2), all and (min--moz-device-pixel-ratio: 1.5), all and (min-device-pixel-ratio: 1.5) { }// After
@media all and (-webkit-min-device-pixel-ratio: 1.5),
all and (-o-min-device-pixel-ratio: 3/2),
all and (min--moz-device-pixel-ratio: 1.5),
all and (min-device-pixel-ratio: 1.5) {
}
Less and Scss are turning into real programming languages :) Step by step, we're starting to print all their constructs in the same way as JavaScript. This time, it's the else
placement.
// Before @if $media == phonePortrait { $k: .15625; } @else if $media == tabletPortrait { $k: .065106; }// After
@if $media == phonePortrait {
$k: .15625;
} @else if $media == tabletPortrait {
$k: .065106;
}
While we want prettier to format the entire codebase, there are times where we "know better" and want an escape hatch. This is where the prettier-ignore
comment comes in. It wasn't working for CSS but that was an oversight, now it is implemented :)
// Before foo { /* prettier-ignore */ thing: foo; -ms-thing: foo; }// After
foo {
/* prettier-ignore */
thing: foo;
-ms-thing: foo;
}
In order to be fast, many "packagers" do not parse files in order to extract dependencies but instead use a crude regex. This is a reason why we don't break long require()
calls and it happens to also affect CSS Modules. If you add new lines in the composes
field, it doesn't recognize it anymore. So we're no longer breaking it there, even if it goes over 80 columns.
// Before .reference { composes: selector from "a/long/file/path/exceeding/the/maximum/length/forcing/a/line-wrap/file.css"; }// After
.reference {
composes: selector from "a/long/file/path/exceeding/the/maximum/length/forcing/a/line-wrap/file.css";
}
We made a decision to have only a single high level "parser" for CSS, SCSS and Less even though we are using postcss-less
and postcss-scss
under the hood. We use a regex to figure out which parser to try first and fallback to the other one if a syntax error is thrown. Unfortunately, for certain features, the first (incorrect) parser doesn't throw and instead skips some elements. So, we need to beef up the regex to make sure we are right for the early detection.
Thankfully, this hack is working well in practice. If we find a lot more edge cases, we'll likely want to do the right thing(tm) and split them into two parsers.
// Before @import "text-shadow";// After
@import "rounded-corners", "text-shadow";
TypeScript support is now solid, all the changes for this release are small edge cases.
The core algorithm of prettier is to expand a group if all the elements do not fit. It works really well in practice for most of JavaScript but there's one case it doesn't handle very well is when there are two groups side by side, in this case: <Generics>(Arguments)
. We have to carefully create groups such that arguments expand first as this is generally what people expect.
// Before export const forwardS = R.curry(< V, T >(prop: string, reducer: ReducerFunction<V, T>, value: V, state: {[name: string]: T}) => R.assoc(prop, reducer(value, state[prop]), state) );// After
export const forwardS = R.curry(
<V, T>(
prop: string,
reducer: ReducerFunction<V, T>,
value: V,
state: { [name: string]: T }
) => R.assoc(prop, reducer(value, state[prop]), state)
);
For better or worse, we decided to manually handle adding parenthesis. So when a new operator is introduced, we need to make sure that we add correct parenthesis when nested with any other combination of operators. In this case, we missed await inside of TypeScript !
.
// Before const bar = await foo(false)!;// After
const bar = (await foo(false))!;
We use typescript-eslint-parser project that translates TypeScript AST into estree AST in order for prettier to print it. From time to time we're going to find edge cases that it doesn't handle yet. In this case, it didn't give a way to tell that there's an empty {}
, which apparently is important for TypeScript. Thankfully, the team is very responsive and they fixed it after we put a workaround inside of prettier.
// Before import from "@types/googlemaps";// After
import {} from "@types/googlemaps";
The code that implements interface
is shared with the code that prints object
s, which contains a rule to keep them expanded if there's a \n
inside. But, this is not the intended behavior for interfaces. We always want to expand, like we do for classes, even if it fits 80 columns.
// Before interface FooBar { [id: string]: number; }// After
interface FooBar {
[id: string]: number;
}
no-semi
and semi
are often requested but on the prettier team we're one step ahead and implemented two-semi
for you! Just kidding, it was a bug and is now fixed ;)
// Before declare module "classnames" { export default function classnames( ...inputs: (string | number | false | object | undefined)[] ): string;; }// After
declare module "classnames" {
export default function classnames(
...inputs: (string | number | false | object | undefined)[]
): string;
}
Adding a comment before a method used to take into account the comment length and would often expand the method when it wasn't expected. Thankfully, it was a simple fix, just wrap the output in a group
.
// Before interface TimerConstructor { // Line-splitting comment new ( interval: number, callback: (handler: Timer) => void ): Timer; }interface TimerConstructor {
// Line-splitting comment
new (interval: number, callback: (handler: Timer) => void): Timer;
}
This bug was very annoying if you ran into it: anytime you formatted the code, it would add one more _
to the object key!
// Before obj = { __: 42 ___: 42 };// After
obj = {
_: 42
__: 42
};
Unlike in JavaScript, TypeScript lets you extend multiple classes at once. It turns out that people use this feature and prettier now does a better job at printing it.
// Before export interface ThirdVeryLongAndBoringInterfaceName extends AVeryLongAndBoringInterfaceName, AnotherVeryLongAndBoringInterfaceName, AThirdVeryLongAndBoringInterfaceName {}// After
export interface ThirdVeryLongAndBoringInterfaceName
extends AVeryLongAndBoringInterfaceName,
AnotherVeryLongAndBoringInterfaceName,
AThirdVeryLongAndBoringInterfaceName {}
This one isn't very interesting, it's an edge case that's not properly handled in the TypeScript -> estree conversion.
// Before call(c => { bla: 1 }) || [];// After
call(c => ({ bla: 1 })) || [];
By supporting TypeScript, prettier is now being used in a lot of Angular codebases which exercises edge cases that were not properly handled. In this case, we didn't preserve empty lines after directives inside of a function.
// Before export default class { constructor($log, $uibModal) { "ngInject"; Object.assign(this, { $log, $uibModal });// After
export default class {
constructor($log, $uibModal) {
"ngInject";<span class="pl-c1">Object</span>.<span class="pl-en">assign</span>(<span class="pl-c1">this</span>, { $log, $uibModal });</pre></div>
JavaScript
This release is very light in terms of JavaScript changes, which is awesome. We're starting to see the light at the end of the tunnel and get towards a great pretty printer. We're never going to get to a 100% perfect automatic pretty printer. The goal is that for every issue we get, there are no clear ways to improve the way it is printed without regressing other pieces.
Allow JSX lines to be recombined (#1831) by @karl
The goal of prettier is to have a consistent way to format your code: given an AST, we always print the same way. In two places we had to compromise and read the original format: JSX and Objects. With this change, we're no longer relying on the original input for JSX with text inside. This lets us reflow text inside of JSX.
// Before const Abc = () => { return ( <div> Please state your {" "} <b>name</b> {" "} and {" "} <b>occupation</b> {" "} for the board of directors. </div> ); };// After
const Abc = () => {
return (
<div>
Please state your <b>name</b> and <b>occupation</b> for the board of
directors.
</div>
);
}Break on non-literal computed member expression (#2087) by @azz
Printing member chains is the most complicated piece of prettier and we keep finding small tweaks we can do to make it a better experience.
// Before nock(/test/) .matchHeader("Accept", "application/json")[httpMethodNock(method)]("/foo") .reply(200, { foo: "bar", });// After
nock(/test/)
.matchHeader("Accept", "application/json")
[httpMethodNock(method)]("/foo")
.reply(200, {
foo: "bar",
});Indent first variable in one-var scenario (#2095) by @azz
Up until recently we haven't done much to support printing multiple variables in a single declaration as the most common practice is to do one variable declaration per variable. For single declarations, we don't want to indent it, but it turns out that we do when there are other ones afterwards, otherwise it looks weird.
// Before var templateTagsMapping = { '%{itemIndex}': 'index', '%{itemContentMetaTextViews}': 'views' }, separator = '<span class="item__content__meta__separator">•</span>';// After
var templateTagsMapping = {
'%{itemIndex}': 'index',
'%{itemContentMetaTextViews}': 'views'
},
separator = '<span class="item__content__meta__separator">•</span>';Allow break with both default named import (#2096) by @azz
This one is an unfortunate regression from 1.4 where we inlined import that only contained a single element. Turns out the definition of a single element allowed a single type and a single element. This is now corrected!
// Before import transformRouterContext, { type TransformedContextRouter } from '../../helpers/transformRouterContext';// After
import transformRouterContext, {
type TransformedContextRouter
} from '../../helpers/transformRouterContext';Turn allowImportExportEverywhere on (#2207) by @zimme
The goal of prettier is to format code people write in practice, so we enable loose/experimental modes for all the parsers we support. Babylon allows you to write import within a function, which is not part of the standard, but it doesn't cost us much to allow it.
// Before ParseError// After
function f() {
import x from 'x';
}Support inline template for new calls (#2222)
We keep adding features for function calls and have to backport them to new calls as they have a different AST node type but in practice we want to treat them the same. This fix refactored the two so that they are going through the same call site, so hopefully should prevent more from sneaking in.
// Before new Error( formatErrorMessage` This a really bad error. Which has more than one line. ` );// After
new Error(formatErrorMessage</span></span> <span class="pl-s"> This a really bad error.</span> <span class="pl-s"> Which has more than one line.</span> <span class="pl-s"><span class="pl-pds">
);Don't indent + in object value (#2227)
When we switched to using the same heuristic for assignment (
a = b
) for objects ({a: b}
), we forgot to fix the indentation. Now it's fixed.// Before var abc = { thing: "asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdf" + "asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdf" + "asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdf", }// After
var abc = {
thing:
"asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdf" +
"asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdf" +
"asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdf",
}Handle conditions inside of a ternary (#2228)
Prettier already had a special case when the expression was a conditional but it didn't apply when the conditional was the left part of a ternary. Now it does.
// Before room = room.map((row, rowIndex) => row.map( (col, colIndex) => rowIndex === 0 || colIndex === 0 || rowIndex === height || colIndex === width ? 1 : 0 ) );// After
room = room.map((row, rowIndex) =>
row.map(
(col, colIndex) =>
rowIndex === 0 ||
colIndex === 0 ||
rowIndex === height ||
colIndex === width
? 1
: 0
)
);Add caching for printing (#2259)
With the 1.0 release, we fixed a bug in the printing that introduced an exponential behavior. We've been able to mitigate the biggest issue such that reasonable code didn't time out, but it wasn't completely fixed it. By adding a caching layer at the right spot, we should now be in the clear.
This should make printing the IR of prettier using prettier in debug mode no longer time out.
// Before ...times out...// After
someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
return someObject.someFunction().then(function () {
anotherFunction();
});
});
});
});
});
});
});
});
});Fix variance location (#2261)
We refactored the code that prints modifiers when we introduced TypeScript support and accidentally moved around the variance (
+
) part beforestatic
which is not valid in Flow. This is now fixed.// Before class Route { +static param: T; }// After
class Route {
static +param: T;
}Miscellaneous
Various fixes for range and cursor tracking (#2266, #2248, #2250, #2136) by @CiGit and @josephfrazier
Both those features were introduced in the last release and we discovered a bunch of issues when actually using them in production. A bunch of them got fixed, if you see more, please report them!
Not sure how things should work exactly?
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot 🌴
Branch | Build failing 🚨 |
---|---|
Dependency | eslint-plugin-jsx-a11y |
Current Version | 5.0.0 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As eslint-plugin-jsx-a11y is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.
I recommend you give this issue a high priority. I’m sure you can resolve this 💪
Swapped Array.includes
for array-includes
polyfill to support node versions <4
The new version differs by 3 commits0.
bf0ac6a
5.0.1
fb8c626
Upgrade all dependencies.
fb5b87a
Fix: backward compatibility with node.js v4 and v5. (#231)
false
See the full diff
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot 🌴
Branch | Build failing 🚨 |
---|---|
Dependency | eslint |
Current Version | 3.14.1 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
As eslint is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.
I recommend you give this issue a high priority. I’m sure you can resolve this 💪
no-extra-parens
incorrect precedence (fixes #7978) (#7999) (alberto)template-tag-spacing
rule (fixes #7631) (#7913) (Jonathan Wilsson)The new version differs by 17 commits .
8ea98f9
3.15.0
77b5cb7
Build: package.json and changelog update for 3.15.0
f2a3580
Fix: no-extra-parens
incorrect precedence (fixes #7978) (#7999)
d6b6ba1
Fix: no-var should fix ForStatement.init (#7993)
99d386d
Upgrade: Espree v3.4.0 (#8019)
42390fd
Docs: update README.md for team (#8016)
d7ffd88
Chore: enable template-tag-spacing on ESLint codebase (#8005)
f2be7e3
Docs: Fix typo in object-curly-newline.md (#8002)
df2351a
Docs: Fix misleading section in brace-style documentation (#7996)
5ae6e00
Chore: avoid unnecessary feature detection for Symbol (#7992)
5d57c57
Chore: fix no-else-return lint error (refs #7986) (#7994)
62fb054
Chore: enable no-else-return on ESLint codebase (#7986)
c59a0ba
Update: add ignoreRestSiblings option to no-unused-vars (#7968)
5cdfa99
Chore: enable no-unneeded-ternary on ESLint codebase (#7987)
fbd7c13
Update: ensure operator-assignment handles exponentiation operators (#7970)
There are 17 commits in total. See the full diff.
There is a collection of frequently asked questions and of course you may always ask my humans.
Your Greenkeeper Bot 🌴
Is there a description for the disallow and crawl delay tags?
I am using disallow: ['/page/*/', '/other-page/*/'],
but I see both of them in the XML when I build.
Have I not understood it correctly?
Hello)
I use Angular 8 and angular universal.
How i must use your library for automatically generate 'robot.txt' file on server build?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.