gadicc / meteor-sitemaps Goto Github PK

Does your sitemap package require Blaze templates and iron router? I tried to use it and had no errors whatsoever, but could not find where it placed the sitemap.xml file. I used the following first few lines to start the process, as you have in your documentation
sitemaps.add('/sitemap.xml', function() {

Maybe I just am not sure where to look for the file.

Thanks

i have exact same issue like this meteor forum post https://forums.meteor.com/t/using-meteor-with-multiple-domains

we have an app for multiple domain name, and the only way to get the domain name is from req.headers.host, can you add in the "req" or "req.headers" object to sitemaps.add function just like the solutions suggested in the forum post?

How to create "sitemap index" by using this package or any of meteor package..??

This is useful in multilingual meteor websites

https://support.google.com/webmasters/answer/2620865?hl=en

Can you please provide support for flow router?
Thanks

Hello,

are your sitemaps working correctly? I just submitted mine generated using this package but I got a google error:

Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.

Hey Gadi,

Your package didn't automatically migrate (actually due to some weirdness at our end, nothing you did I don't think).

Any way, it should be super easy to upload a new working 0.0.14 version to the package server! Instructions here: https://hackpad.com/Migrating-Packages-zN0we9sIjkH

And remove me as a maintainer :)

Is this suitable to use with fairly bigger sitemap - 50,000 results? Recently I started to get 502 error when trying to access .xml file for this sitemap.

FATAL ERROR: Evacuation Allocation failed - process out of memory [192.81.217.53] error: Forever detected script was killed by signal: SIGABRT [192.81.217.53] error: Script restart attempt #11[192.81.217.53]

I've bundled it and it's live:
http://coderstv.com/sitemap.xml

What can I do ?

How to do dynamic sitemaps when I have something like this with Iron router:

this.route('/analytics/:_id/:name', {
   template: 'analyticsDetails',
    waitOn: function() {
        return Meteor.subscribe('Analytics', {
            _id: this.params._id
        })
    },
    data: function() {
        return Analytics.findOne(this.params._id);
    },
    onAfterAction: function () {
        SEO.set({
            title: this.params.name + " | GrowthTools.io",
            description: this.params.functionality
        });
    }
})

I'm using video tag. When I add thumbnail_loc/content_loc with absolute CDN url, it escaped and added root url.

how to import with meteor 1.3 ?
i tried import 'meteor/gadicohen:sitemaps';

and error:

ReferenceError: sitemaps is not defined

Please @gadicc can you update the README.md file to make it obvious that the .xml files are created on-the-fly during the request by the crawler? Thanks

Hi!

I just tested my sitemap in Webmaster Tools and it gave this error:

Unsupported file format: Your Sitemap does not appear to be in a supported format. Please ensure it meets our Sitemap guidelines and resubmit.

I just have a single sitemaps.add statement that adds all my publicly-accessible routes, and the generated file looks correct to me (you can see it here: https://modtest.canlio.com/sitemap.xml).

Any ideas on what I'm doing wrong?

hello @gadicc

updated my project to meteor 0.8. I appear to be having some issues with this sitemaps package.

I use the following code in my sitemap code to generate dynamic post for indexing:

var stories = Stories.find().fetch();
  _.each(stories, function(story){
    out.push({
      page: '/story/'+story._id+'/'+encodeURIComponent(story.title),
      lastmod: new Date().getTime(),
      changefreq: 'daily',
      priority: 0.6
    })
  });

But then I got this error server side:

W20140407-15:08:41.372(8)? (STDERR) Error: Can't wait without a fiber
W20140407-15:08:41.372(8)? (STDERR)     at Function.wait (/Users/shawnlim/.meteor/tools/c2a0453c51/lib/node_modules/fibers/future.js:83:9)
W20140407-15:08:41.372(8)? (STDERR)     at Object.Future.wait (/Users/shawnlim/.meteor/tools/c2a0453c51/lib/node_modules/fibers/future.js:325:10)
W20140407-15:08:41.372(8)? (STDERR)     at _.extend._nextObject (packages/mongo-livedata/mongo_driver.js:805)
W20140407-15:08:41.372(8)? (STDERR)     at _.extend.forEach (packages/mongo-livedata/mongo_driver.js:836)
W20140407-15:08:41.372(8)? (STDERR)     at _.extend.map (packages/mongo-livedata/mongo_driver.js:846)
W20140407-15:08:41.372(8)? (STDERR)     at _.extend.fetch (packages/mongo-livedata/mongo_driver.js:870)
W20140407-15:08:41.373(8)? (STDERR)     at Cursor.(anonymous function) [as fetch] (packages/mongo-livedata/mongo_driver.js:695)
W20140407-15:08:41.373(8)? (STDERR)     at app/server/sitemaps.js:7:32
W20140407-15:08:41.373(8)? (STDERR)     at Object.sitemaps.add.root [as handle] (packages/sitemaps/sitemaps.js:37)
W20140407-15:08:41.373(8)? (STDERR)     at next (/Users/shawnlim/.meteor/packages/webapp/9dd975a6fc/npm/node_modules/connect/lib/proto.js:190:15)

Maybe meteor adjusted a dependency you were using?

Hi, I'm trying to generate a sitemap. I have a collection "Quicks" that contains about a handful of search shortcuts. Each document of "Quicks" is rendered as an URL in the sitemap. For each I'd like to get the date of the last upload (to the collection "EboyPix") — and use it as the lastmod property for the sitemap.

When iterating through "Quicks" the first two return undefined for the createdAt field. As if function "getLatestQuickDate" needed a bit of time to get going.

What can I do? Thanks!

/server/sitemap.js

// Meteor
import { sitemaps } from 'meteor/gadicohen:sitemaps';

// Collections
import { Quicks } from '../imports/api/quicks/quicks.js'; // contains search shortcuts
import { EboyPix } from '../imports/api/eboypix/eboypix.js';

const quicksLabels = Quicks.find().fetch();

// Returns the latest date from the slug
function getLatestQuickDate(slug) {
  const latestSlug = EboyPix.findOne(
    { tags: slug },
    { sort: { createdAt: -1, limit: 1 } }
  );
  if (latestSlug) {
    return latestSlug.createdAt;
  }
}

// Returns array of pages from Quicks collection
const quicksLinks = function() {
  let quicksPagesAll = [];
  if (quicksLabels) {
    Object.keys(quicksLabels).forEach(function(key) {
      const slug = quicksLabels[key].slug.toString();
      const lastmod = getLatestQuickDate(slug);
      const quicksPage = {
        page: 'pool/' + quicksLabels[key].label + '/1',
        changefreq: 'weekly',
        priority: quicksLabels[key].rank,
        lastmod: lastmod
      };
      quicksPagesAll.push(quicksPage);
    });
    return quicksPagesAll;
  }
};

sitemaps.add('/sitemap.xml', function() {
  return quicksLinks();
});

output: /sitemap.xml

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>http://localhost:3000/pool/everything/1</loc>
    <changefreq>weekly</changefreq>
    <priority>1</priority>
  </url>
  <url>
    <loc>http://localhost:3000/pool/pixorama/1</loc>
    <changefreq>weekly</changefreq>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>http://localhost:3000/pool/animations/1</loc>
    <lastmod>2017-02-24T08:22:37+00:00</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.1</priority>
  </url>
  <url>
    <loc>http://localhost:3000/pool/vehicles/1</loc>
    <lastmod>2017-02-02T08:53:21+00:00</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.5</priority>
  </url>
  <url>
    <loc>http://localhost:3000/pool/portraits/1</loc>
    <lastmod>2017-02-02T21:57:44+00:00</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.7</priority>
  </url>
</urlset>

Hi there!
I'm trying to apply sitemaps package functionality using next code within /server/sitemaps.jsx file:

sitemaps.config('gzip', true);
sitemaps.add('/sitemap.xml.gz', function(req) {
  console.log('Sitemap created')
  const out = [] 
  const ads = Ads.find().fetch()
  ads.forEach(ad => {
    out.push({
      ad: ad.url,
      lastmod: ad.created
    });
  });
  return out;
});

But I see that console.log on the 3d line doesn't appear. Does that mean that a gz file is not created and thus callback is not called?
What could be a problem? Permissions?
Thanks!

Hey

I try your package but it seems not to be work. This is my file:

server/sitemaps.js

console.log('Adding to Sitemap');
sitemaps.add('/sitemap.xml', function() {
  return [
    { page: '/user/login', lastmod: new Date() , changefreq: 'monthly', priority: 0.8}
  ];
});

This do .... nothing. Absolutly nothing. My console.log is outputted in my server log, but no sitemap.xml was created. I tried to add a file by my self - a sitemap.xml file - but this will also not be updated with this package.

I read, that you add a line to robots.txt when using robots.txt package - but I do not use this and I dont want to - so do I need to use this package so your will work ?

CPU usage 100% with this error on DigitalOcean

    at errnoException (child_process.js:1011:11)
    at ChildProcess.spawn (child_process.js:958:11)
    at exports.spawn (child_process.js:746:9)
    at Object.exports.execFile (child_process.js:628:15)
    at Object.Package [as handle] (packages/ongoworks_spiderable/packages/ongoworks_spiderable.js:141:1)
    at next (/opt/meteor/app/programs/server/npm/webapp/node_modules/connect/lib/proto.js:190:15)
    at packages/gadicohen_sitemaps/packages/gadicohen_sitemaps.js:71:1
Error: spawn ENOMEM
    at errnoException (child_process.js:1011:11)
    at ChildProcess.spawn (child_process.js:958:11)
    at exports.spawn (child_process.js:746:9)
    at Object.exports.execFile (child_process.js:628:15)
    at Object.Package [as handle] (packages/ongoworks_spiderable/packages/ongoworks_spiderable.js:141:1)
    at next (/opt/meteor/app/programs/server/npm/webapp/node_modules/connect/lib/proto.js:190:15)
    at packages/gadicohen_sitemaps/packages/gadicohen_sitemaps.js:71:1
Error: spawn ENOMEM
    at errnoException (child_process.js:1011:11)
    at ChildProcess.spawn (child_process.js:9[IP] 58:11)
    at exports.spawn (child_process.js:746:9)
    at Object.exports.execFile (child_process.js:628:15)
    at Object.Package [as handle] (packages/ongoworks_spiderable/packages/ongoworks_spiderable.js:141:1)
    at next (/opt/meteor/app/programs/server/npm/webapp/node_modules/connect/lib/proto.js:190:15)
    at packages/gadicohen_sitemaps/packages/gadicohen_sitemaps.js:71:1
spiderable: phantomjs failed: { [Error: Command failed: ] killed: true, code: null, signal: 'SIGTERM' } 

Add support for site maps like https://yourdomain.com/sitemap.xml

Hi, Thanks for this wonderful and helpful package. Works well.
There is this issue i get grom Google's webmaster's console. It says

Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.
though the sitemap's url is /sitemap.xml here's the url naijapoll.com/sitemap.xml

We create an sitemap indexes dynamically for companies by countries.

<sitemap><loc>http://example.com/sitemaps/SG/companies.xml<loc></sitemap>
<sitemap><loc>http://example.com/sitemaps/US/companies.xml<loc></sitemap>
<sitemap><loc>http://example.com/sitemaps/MY/companies.xml<loc></sitemap>
                            ...

So is it possible to implement Path-to-RegExp to sitemaps url? example:

sitemaps.add("/sitemaps/:country/companies.xml", function(params){
    var paths = []
    Company.find({'iso3':params.country}).forEach(function(doc){
        paths.push({
            page: "/company/" + doc.slug
        })
    })
    return paths;
})

Yop,

I would like add your module on my Meteor App, but /sitemap.xml return this error :

TypeError: Cannot read property 'find' of undefined
   at imports/startup/server/sitemaps.js:10:17
   at packages/gadicohen_sitemaps.js:120:15 

I use latest Meteor version (5).

My sitemaps.js (server) :

import { Meteor } from 'meteor/meteor';
import { sitemaps } from 'meteor/gadicohen:sitemaps';
import { Post } from '../../api/Post.js';

sitemaps.config('rootUrl', 'https://stephane-richin.fr/');

sitemaps.add('/sitemap.xml', () => {

  const postsUrls = [];
  const posts = Post.find({ 'draft': false }).fetch();

  _.each(posts, function (page) {

    postsUrls.push({
      page: `/blog/${page.slug}`,
      lastmod: new Date()
    });

  });

  const staticPages = [
    { page: '/', lastmod: new Date() },
    { page: '/blog', lastmod: new Date() },
    { page: '/a-propos', lastmod: new Date() },
    { page: '/contact', lastmod: new Date() }
  ];

  const urls = [...staticPages, ...postsUrls];

  // required: page
  // optional: lastmod, changefreq, priority, xhtmlLinks, images, videos
  return urls;

});

Do you have any idea ?

Thank you :)

@gadicc Hello, Does your package support the new version of meteor?
And is it up to date with the Google update?
If you no longer support this package. Please let users know

I have pages that live one month. And for good seo I need to remove they from sitemap. Any advice how to do it ?

Hello and thank you for this package.

I noticed something interesting while I was playing with it. When we add alternate urls, the rendering of the XML document in the browser breaks.

Normal Behavior. (without alternate urls)

sitemaps.add '/sitemap.xml', ->
  return [
    { page: '/', changefreq: 'weekly' }
    { page: '/manifesto', changefreq: 'weekly' }
    { page: '/contact', changefreq: 'weekly' }
  ]

Broken Rendering. (with alternate urls)

sitemaps.add '/sitemap.xml', ->
  return [
    { page: '/', changefreq: 'weekly', xhtmlLinks: [
      { rel: 'alternate', hreflang: 'en', href: '/en' },
      { rel: 'alternate', hreflang: 'fr', href: '/fr' },
      { rel: 'alternate', hreflang: 'es', href: '/es' }
    ]}
    { page: '/manifesto', changefreq: 'weekly' }
    { page: '/contact', changefreq: 'weekly' }
  ]

# It seems that adding the following lines break the rendering process.
<url>
    <loc>http://localhost:3000/</loc>
    <changefreq>weekly</changefreq>
    <xhtml:link rel="alternate" hreflang="en" href="http://localhost:3000/en"/>
    <xhtml:link rel="alternate" hreflang="fr" href="http://localhost:3000/fr"/>
    <xhtml:link rel="alternate" hreflang="es" href="http://localhost:3000/es"/>
</url>

# but this is perfectly rendering.
<url>
    <loc>http://localhost:3000/</loc>
    <changefreq>weekly</changefreq>
</url>

We're wondering when the sitemap is actually created, because according to the docs, it doesn't appear that we manually call it anywhere, it just runs at some time. When does it run?

If it does run more than one time, it should duplicate the sitemap but the duplicate will have the updated collection info, which is something we need. Our collection contains ~17,000 companies and they change over time so we would like our sitemap to be updated periodically, say about once a month.

Also, somewhat related is, if it runs on some regular basis, does it delete and recreate the sitemap each time or will we need to manually remove it every time it runs?

Is sitemap change, if db collection changed (which use to sitemap)?
For example, is sitemap update, if WikiPages was updated?

sitemaps.add('/mw_AllPages_sitemap.xml.gz', function(req) { var out = [], pages = WikiPages.find().fetch(); _.each(pages, function(page) { out.push({ page: 'read/' + page.name, lastmod: page.lastUpdated }); }); return out; });

Well it work to add * or ? link in robot.txt file

{ page: '/public*', lastmod: new Date() },

basiclly want all pages with public to be created

I'd love to help implement gzip (see here). How can we proceed?

Navigating to /sitemap.xml results in a redirect loop between /#!sitemap.xml and /sitemap.xml.

Not sure if this should be considered an iron:router bug or a sitemaps bug.

Any thoughts, on what to check or where the issue may be?

Hi,
First thanks for your awesome package.
I try to use your package but my sitemap.xml shown this error in browser:

XML Parsing Error: not well-formed
Location: http://localhost:3000/sitemap.xml
Line Number 22, Column 60:    <loc>http://localhost:3000/pages/title?q=query&lang=</loc>
---------------------------------------------------------------------------------^

But my sitemap seems to work. When viewing the source of my sitemap, the XML needed for crawlers is displayed.

URL is: http://localhost:3000/pages/title?q=query&lang=

How to add escape special character like this.

Is it possible to add image:image tag ?

Hi
I recently tried to use the pathFor in a sitemap function with a trick like this:

var pathFor = UI._globalHelpers.pathFor;
out.push({
        page: pathFor('home'),
        lastmod: new Date(2015, 0, 1),
        changefreq: 'daily',
        priority: 0.5
});

But i get the error in console saying

(STDERR) pathFor couldn't find a route named undefined

Is there a proper way for this?

Currently, sitemaps' URLs are all based on Meteor.absoluteUrl(), which in turn draws from the ROOT_URL environment variable. Over at TelescopeJS/Telescope#891, we're trying to allow sites' root URLs to be user-configurable (using mongo-stored settings), so we need a way to set the root URL that doesn't involve changing environment variables.

Would you accept a PR to take a third argument on sitemaps.add which optionally specifies the root URL?

Hello (again) Gadi!

Looks like I'm bumping into your really handy packages this week :-)

I've got a simple site for my wife's bed and breakfast which uses your meteor-headers package to figure out the language and than redirects to the language specific section:

/ -> /en and in case of Dutch / -> /nl

As such, the home is never really used since there's always the redirect to /lang

More languages will follow.

What do you think of this approach?

I'd like to configure this in the sitemap but being a n00b here maybe you can point me in the right direction?

Kind thanks,

. Dirk

Great module!

You might consider adding the ability to use an XSL stylesheet such as Yoast's: https://yoast.com/xsl-stylesheet-xml-sitemap/

One idea would be to include a default stylesheet &/or to allow linking to a stylesheet that is stored in an app's public directory.