sensiblecodeio / custard Goto Github PK

Hide it for now, until it does something.

Visiting something made up like:

http://localhost:3001/set-password/8166madeuptoken7615

Causes Custard to crash, because it's searching the model for a matching token, and when it doesn't find one, the error isn't handled.

If I run a Python script like this over the exec endpoint, with "python -u" to stop Python doing any buffering:

x = 0
while x < 1000:
    x = x + 1
    print "hello", x
    time.sleep(0.01)

And look in the browser network navigation, I find the data is arriving in 4Kb chunks.

I suspect this is nginx buffering it:
http://blog.jamieisaacs.com/2010/08/27/comet-with-nginx-and-jquery/

Similar to exec, only it would call the sql endpoint.

In particular "view data" at the top.

While we have few tools, we can just curate this order for everyone.

e.g. when I had broken mongodb entry for my username (because chris edited API key at same time I changed password), and it was crashing during login.

All I got was a "bad gateway" error.

As a user, I would expect some kind of error message (doesn't have to be pretty right now - but something saying it is just an error in the web app code, rather than the whole site is broken which bad gateway looks like).

As a developer, I'd expect logs on the servers with the error stack trace in etc.

Both for its name in github "spreadsheet-tool", and on the button to use it.

spreadsheet-view-tool would be a better repository name

And in use the button should be something more like "View data" or "View table of data"

I was editing a short default name to "People who responded to ScraperWiki survey", and it had to scroll.

Could have just a longer minimum default when editing the name. (Or more fancily, have it expand as you type).

I got this error "Oh no! We received an error from the extraction script: The column name must be of unicode or str type. The column name ("41035.9381944") is of type <type 'float'>. If this error doesn't make sense, try "unicode('41035.9381944')".

I think it should show the erro from the extraction script in a technical section formatted differently below.

The main error should say that it couldn't import it, and clearly explain what kind of spreadsheets it can import. e.g. They must have one table at most in each tab, with a clear heading.

When deleting a dataset from 'My datasets' by clicking the x I don't get a second chance, or warning about deleting data, or an "Are you sure?".

Would be nice to have an undo.

I made a couple of Highrise datasets I don't care about, and it's really annoying clutter.

I'd like some way of deleting them. (I don't care much for now if all it does is hide them from the user interface)

In every other website, if I press Cmd and click it opens in a new window.

This doesn't work on x.scraperwiki.com. If i right click and do "open in new tab" it is fine.

Now we're using easyXDM and nested iframes, the scroll bars and padding on views aren't right.

On the view source page, if I've already added an SSH key the wording is wrong.

Perhaps generalise the wording, if detecting it is too hard

I did this:
scraperwiki.tool.redirect("https://api.twitter.com/oauth/authorize?oauth_token=....")

And it redirected me to:
http://x.scraperwiki.com/https://api.twitter.com/oauth/authorize?oauth_token=....

When i click the "About ScraperWiki" link the left-hand navigation panel, I get a picture of a pig.

I am disappoint.

(the link in the A element looks correct, it must be some funky push-state URL handling code)

When you make a dataset (e.g. "highrise") the default names should be written in sentence case with spaces, e.g. "Highrise", "New dataset"

Do not want.

I've been actively using x.scraperwiki.com, nevertheless I still have to login each day. Right now it has prompted me to login, and my cookie is like this:

x.scraperwiki.com
Name: connect.sid
Content: s%3AsHuOccFb5vY0GMOolIwpBL24.zwwlocaLPltUfGseKasfdFsi%2BgIUNzOx4n4ohfsXZWE
Domain: x.scraperwiki.com
Path: /
Send for: Any kind of connection
Accessible to script: No (HttpOnly)
Created: Wednesday, February 6, 2013 10:48:06 AM
Expires: Thursday, February 7, 2013 10:48:03 AM

I'm on latest stable Chrome on OSX.

If you try and search in it, it can't find anyone which is very confusing.

It should just not show if you are not an admin user.

There's no way of reusing common html elements (eg: the SSH instructions that are shown by the newdataset-tool, newview-tool and view-source-tool).

Rather than HTTP.

They end up called "Upload a spreadsheet".

Ideally, I think they should have some kind of title from the Excel file. Or failing that (no title, CSV file) the filename.

e.g. "Spreadsheet Tool!" vs. "spreadsheet".

It's very unclear what the relationship between them is.

When I go to /tools and choose "Code a dataset" it takes me to a new page /dataset/cl3jyki/settings which is blank. I have the following errors in my console

POST http://boxecutor-live-0.scraperwiki.net/cl3jyki/exec 403 (Forbidden) jquery.min.js:2
POST http://boxecutor-live-0.scraperwiki.net/box/dnbl2ni 403 (Forbidden) jquery.min.js:2
POST http://boxecutor-live-0.scraperwiki.net/dnbl2ni/exec 403 (Forbidden) jquery.min.js:2

Presumably the follow on errors when I go to My Datasets and select the dataset followed by Code my own view are related, they're:

POST http://boxecutor-live-0.scraperwiki.net/hhqajry/exec 403 (Forbidden) jquery.min.js:2
POST http://boxecutor-live-0.scraperwiki.net/box/elxposa 403 (Forbidden) jquery.min.js:2
POST http://boxecutor-live-0.scraperwiki.net/elxposa/exec 403 (Forbidden)

IDs in examples will be different because I tried this a few times ;)

I had some Pocket data, and it was at top right left of my other three datasets.

Then I added a View to the Pocket dataset. It jumped in "My Datasets" to appear at the top right, right of my other three datasets.

I expect it to stay in a consistent order when I add a view.

When on a dataset page (e.g. http://x.scraperwiki.com/dataset/ddjgwgy) the blue box representing the dataset itself changes cover when you hover over it.

This makes it look like it is clickable, when in this context it isn't.

When I click the scratch marks on the bottom of the navigation bar, the one that collapses/expands the navigation bar, the URL is reset to the homepage, and so is the page contents.

Would be good to have some kind of standard for this (e.g. success outputs JSON without an error key, failure is any other output, or whatever), and a simple exec wrapper which displays nice errors for you, and calls a passed in function on success.

There are lots of cases where you don't expect exec to fail, and just want to quickly call it.

The blank dataset tool has an index.html with its own readSettings, when that is now available from scraperwiki.js

https://github.com/scraperwiki/newdataset-tool/blob/master/http/index.html

When visiting /tools I get a list of 3 tools (Highrise, Code a dataset, and Upload a spreadsheet). I've no idea what Highrise does. Would be nice to have a brief description of what the tool does.

It would be lovely if it did, and encourage people to run scripts by doing tool/foo.py (rather than cd /home/tool; ./foo.py which puts scraperwiki.sqlite in the wrong place with default Dumptruck settings)

When you successfully import a spreadsheet it just says 'Spreadsheet imported! Please navigate to your dataset somehow. (Yes, this needs to be improved)"

Just improving the message to clearly tell you where to navigate to find it might be better!

Not sure what it should be called.

It would mean that there's been no error, but the dataset isn't fully updated yet. e.g. while crawling a backlog of data (due to large volumes in big scrapers, or due to API rate limiting)

Not completely sure if it is needed, but Zarino mentioned it earlier, and I vaguely felt a need for it just now in the Twitter scraper.

Arguably, it is an error if a backlog hasn't been caught up with!

When (iframed) views are passed a URL-encoded JSON object of settings, it should contain publish_token key, not a token.

/home/.git and /home/newdataset/.git

This is confusing!

It was confusing to me for there to be two files both called scraperwiki.json - one for the box and one for the tool manifest.

Not sure what to do about it! Maybe rename the box one?

Click your mugshot in top right, click name in drop down box - raw value of /switch/ is seen.

It'll speed the tests up and stop them breaking when previous tests change.

It'll require everyone to have a local version of mongo on their machines though.

When I'm on the settings page, I tried clicking on the dataset name in the breadcrumb trail to go back to the dataset page. This let you edit the name...

I'd expect it to have an extra "settings" entry at the end of the breadcrumb, and the dataset name take me back to the dataset page.

Make a view, go to it. SSH into it and wipe the top level "scraperwiki.json" file (the one with the database and publish token in it).

The screen is just blank - it should show an error message.

The Javascript error behind the scenes is no more helpful:
Uncaught SyntaxError: Unexpected token c app-c5e5898bdef82296875b73146d71fe1b.js:1
(anonymous function) app-c5e5898bdef82296875b73146d71fe1b.js:1
l jquery.min.js:2
c.fireWith jquery.min.js:2
T jquery.min.js:2
r jquery.min.js:2

When I use the "Code a Dataset!" tool, the dataset it creates should be called "New dataset" or something, not "Code a Dataset!"

It's confusing when the tool, and the thing it creates, both have the same name.

Just tweaking icon so it shows if it had none or lots of data might help scan for datasets.

Need to configure nginx not to cache things.

The settings.json exec response isn't getting parsed properly, I'm not sure why yet.

Valid types are "ok" and "error". At the moment, anything else defaults to "ok".

I think it should be an error to give other types.

(As an aside, I think a blank message is acceptable, the user interface should be good at displaying well if it has a blank message)

Am guessing this will be fixed with view source tool.

Feels strange if that tool is different from the default page help for New Dataset...

Just seeing the random letters for the box is quite confusing. When I look through my terminal windows I'm not confident I'm in the right box.

Would like something in the prompt, or at least in MOTD when logging in, to confirm which dataset it is by showing the dataset name in full.

When you first make a dataset the browser goes to /tool/NAMEOFDATASET

When you go via "my datasets" it is /dataset/NAMEOFDATASET

Leaves me feeling insecure about what the URL is for sharing etc.