atoponce / d-note Goto Github PK

A backend flat file DB (hashcash.db) needs to be installed to keep track of Hashcash tokens created when submitting the note to prevent double spending. See http://hashcash.org/docs/hashcash.html#double_spending_protection. Default expiry of the tokens should be the same as the default expiry of the note, which will likely be the recommendation of 28 days, after which the token will be considered invalid.

The application should be able to handle Unicode input. Currently, it throws the following error:

  File "/usr/share/pyshared/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/share/pyshared/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/share/pyshared/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/pyshared/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/share/pyshared/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/share/pyshared/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/pyshared/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/share/pyshared/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/aaron/src/d-note/d-note.py", line 129, in show_post
    note_encrypt(key, plaintext, new_url, key_file)
  File "/home/aaron/src/d-note/d-note.py", line 68, in note_encrypt
    plain = pad(zlib.compress(plaintext))
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 202: ordinal not in range(128)

Allow the possibility for the note to be viewed multiple times, rather than just once. Default to only one viewing, and give the option for the person submitting the note, to change it. Could be useful for -ops IRC channels, allowing all members to view the note securely, and quickly.

This could be done by maybe requiring usernames/emails to view the note, and force them to provide their username/email to see the note. Open to suggestions.

Possibly provide a warning that multiple views could open up the possibility of others seeing the note unintentionally.

As it currently stands, the cleanup_unread function will destroy the hashcash.db file. Logic needs to be added that prevents this from happening.

Each account should allow the user to create an "address book" for the server to automatically send the note to via SMTP. The administrator would need to add outgoing SMTP support to their server.

The salts should be moved to an external config that is not clobbered by a "git pull" when updating the source. This should free up administrators changing the salts once, and leaving them be, while still being able to update their local repository.

Add the ability to drop a private note into an account "inbox". For those with accounts, could login, and see any unread notes sent to them from others.

Under the "Configuration" Section:
You reference the file named: dnote.ini, however the file must be named: d-note.ini.

The paragraph:

Edit the dnote.ini in this directory and update the 'config_path' value in the [default] section.
It is recommended that this value be either either /etc/dnote or ~/.dnote. 
Once edited, copy the dnote.ini file to the directory you created.

should read:

Edit the d-note.ini in this directory and update the 'config_path' value in the [default] section. 
It is recommended that this value be either either /etc/dnote or ~/.dnote. 
Once edited, copy the d-note.ini file to the directory you created.

Under the "NGINX Setup" Section:
The "module" setting in the UWSGI configuration file reads:
module = __init__:dnote
it should read:
module = __init__:DNOTE

[uwsgi]
socket = 127.0.0.1:8081
chdir = /python/path/site-packages/dnote-1.0.1-py2.7.egg/dnote
plugin = python
module = __init__:DNOTE
processes = 4
threads = 2
stats = 127.0.0.1:9192
uid = www-data
gid = www-data
logto = /var/log/dnote.log

The dconfig.py file is not created with a new install.

Consider relicensing as AGPL v3 http://www.gnu.org/licenses/agpl-3.0.html , as this will provide a network service

When notes are purged, they should be securely scrubbed before deleting from disk. A triple pass of random data over the encrypted note should be more than sufficient, and because the notes will not likely be too large, this shouldn't tax the system too badly.

Each note should allow the user to password protect the note. This will allow the note to be encrypted with AES by using the password as the key, rather than the shared secret. The password will not be stored anywhere on disk, as such, the note will not be recoverable should the password be lost.

In the install.md the Apache setup file creates an Syntax Error.

This line right here
<Directory /var/www/>
Options -Indexes FollowSymLinks

The FollowSymLinks needs to have a + or - in front of it.

Load a separate CSS stylesheet for mobile devices, making it more user-friendly.

Rather than a static 3 minutes, allow users to set a custom redirection timeout.

The timeout could also be set dynamically based on the size of the plaintext. Current reading speeds are about 250 WPM with 70% comprehension average.

The timeout could be encrypted with the plaintext, and when decrypted, sent to the template, or could be added as a URL addition with something like "?t=300" for 300 seconds.

Currently, it's not functioning like it should.

Rudimentary account support needs to be considered. Something like using the /etc/passwd and /etc/shadow files on UNIX systems. But, do flat file account storages scale? LDAP support could be considered.

An administrator account could list current accounts, number of encrypted notes in the DB, IP addresses creating notes, spam prevention, logging, and other stuffs. Reading notes, at least in the web interface however, should probably not be allowed.

Hi!
I (finally) managed to install the app and I'd like the thank some of the issues reported as it helped me to fix my issues.
I am not trying to change a little bit the layout and have edited the base.html, added a custom.css and so on. Unfortunately the web page served is unchanged! I really can't figure this one out.
Does anyone have a clue?
Thank you.
Fred

I'm running d-note on CentOS 7 using uwsgi Emperor. I get the following exception in the /var/log/dnote.log:

[pid: 8395|app: 0|req: 13/58] 10.30.82.185 () {44 vars in 732 bytes} [Tue Mar 17 08:45:58 2015] GET /dnote/ => generated 4990 bytes in 386 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 1)
ERROR:dnote:Exception on /on.ico [GET]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/srv/http/dnote/dnote/__init__.py", line 62, in fetch_url
    note = Note(random_url)
  File "/srv/http/dnote/dnote/note.py", line 68, in __init__
    self.decode_url(url)
  File "/srv/http/dnote/dnote/note.py", line 111, in decode_url
    self.nonce = base64.urlsafe_b64decode(url.encode("utf-8"))
  File "/usr/lib64/python2.7/base64.py", line 112, in urlsafe_b64decode
    return b64decode(s, '-_')
  File "/usr/lib64/python2.7/base64.py", line 76, in b64decode
    raise TypeError(msg)
TypeError: Incorrect padding

Suppose the link to your secret note is https://ae7.st/d/pAJj58F5ozdoEz_w-fEBdw. If there is a typo in the link, such as https://ae7.st/d/pAJj58F5ozdoEz_w-fEBdW, it properly gives a 404. However, if the URL ID is too short, such as https://ae7.st/d/pAJj58F5ozdoEz_w-fEBd or too long such as https://ae7.st/d/pAJj58F5ozdoEz_w-fEBdwwww, it gives a 500 error. It should properly raise a 404, if the URL does not exist, regardless of length.

The web UI will allow a client-generated private id to be used (by altering the value of the hidden new_url input), which allows a malicious user to view and/or modify a note without detection.

After viewing the note, the malicious user only needs to re-create it using the same private id and (optional) passphrase. The note could be identical or modified, and there is no way for the intended recipient to detect this. A note can also be overwritten without being viewed first if the private id is known.

I recommend using a server-side secret to validate that the private id was generated on the server and not by a malicious user. It would also be a good idea to include a timestamp to validate the private id was generated recently.

Something is wrong with the @async decorator. Files older than 30 days are not getting removed.

Whenever a user tries to paste a one-time link into any of a number of communication apps (Facebook, Signal, others) the application "helpfully" tries to read the URL to create a preview. Which sends the message to the shredder.

I don't know if it's possible to identify link preview attempts as such, and return some sort of alternate data (that ideally is agnostic about whether the requested page exists).

Another approach might be to modify the recipient's landing page so that it only indicates that a message is ready to be viewed (or, more agnostically, "here's a code you can try, maybe there's a message associated with it"), but the recipient has to click to actually open and read the message.

An advantage to the second approach is that the landing page has the opportunity to educate the reader about the ephemeral nature of the message they're about to request, before the countdown timer starts.

For context, I use the installation at secrets.xmission.com, and I'm not sure how up-to-date it is.

Thoughts?

Being a Python web application, it should be platform agnostic, anywhere Python is installed. As such, it should be able to store the raw bytes of the AES encryption, rather than worrying about storing the encrypted string in base-64. This would take into account endianess. However, will it scale?

The application will likely need a clean and simple web installer for setting up the environment. Features to include could be:

• SMTP support for outgoing notes to email accounts.
• Generating the shared secret for all notes.
• The expected URL that the application will be installed under.

When at the /post URL, iOS cannot select the full private URL provided in the text box, if the URI contains underscores. Currently, this can only be replicated in Chome for iOS, so it may also be a Chrome bug.

As such, should the base64 characters change in the URL that prevents iOS from doing word separation on underscores?

Is this project still maintained?

The first thing to say is that this is very likely some kind of config mistake that I've made, not a bug in d-note. I can't figure out what's wrong, so I am hoping I can find some troubleshooting help here, so I can figure out what the problem is and get it solved.

I built a replacement certificate (letsencrypt) for my software, installed Ubuntu updates, and rebooted the server. When it came back up, I got timeout issues on the monitoring I've set up, and found that d-note is taking longer than 12 seconds to respond. Everything was fine before the maintenance -- response time was about a tenth of a second.

I have haproxy sitting in front of Apache. The haproxy log seems to indicate that the backend connection to apache is where all the time is being taken. Both haproxy and apache are running https for this virtualhost ... I believe that I tried a non-encrypted apache virtualhost and had problems, so made it all encrypted.

I installed d-note on November 15, 2014. It would have been whatever version was current at that time.

My install is online at https://secrets.elyograg.org where you can see that it's really slow.

Hi,
i have a problem, dnote doesn't work.
i have a error message log
Traceback (most recent call last):
[Mon Jun 06 15:24:47 2016] [error] [client ] File "/var/www/pwgen/dnote.wsgi", line 5, in
[Mon Jun 06 15:24:47 2016] [error] [client ] from dnote import DNOTE as application
[Mon Jun 06 15:24:47 2016] [error] [client ] File "/usr/local/lib/python2.7/dist-packages/dnote-1.0.1-py2.7.egg/dnote/init.py", line 3, in
[Mon Jun 06 15:24:47 2016] [error] [client ] import utils
[Mon Jun 06 15:24:47 2016] [error] [client ] File "/usr/local/lib/python2.7/dist-packages/dnote-1.0.1-py2.7.egg/dnote/utils.py", line 4, in
[Mon Jun 06 15:24:47 2016] [error] [client ] from note import data_dir
[Mon Jun 06 15:24:47 2016] [error] [client ] File "/usr/local/lib/python2.7/dist-packages/dnote-1.0.1-py2.7.egg/dnote/note.py", line 47, in
[Mon Jun 06 15:24:47 2016] [error] [client ] os.sys.exit(1)
[Mon Jun 06 15:24:47 2016] [error] [client ] SystemExit: 1

Can you help me ?
Thanks

Beginning discussion about creating an API here. It should be a RESTful API, as standard. The base API URL would be https://example.com/api/. So far, I've only come up with the basics:

Creation

Creating a note would mean including the required note, the required hashcash token, an optional passphrase, and an optional duress key. As standard, the application will return the random URL. The data in the note could be JSON formatted data, unless there is an easier format to store the data in:

note = [
    {
        'data': u'Encrypt all the things.',
        'hashcash': u'1:20:140615:token::H4pKzifj5g65gx9g:Bzlg', 
        'passphrase': u'Y6stJWMfaUT5E3L4KeSetezQ',
        'duress': u'65y48qszwCmNU3BPGt1U8S3L'
    }
]

The data should also accept a plaintext file to encrypt. The hashcash token will need to be minted either by the application, or by hand using the hashcash(1) utility on Unix. The passphrase should always be generated client-side, so the server knows nothing of it, thus protecting the server administrator (and the end user). The duress key should also be generated client-side. Both the passphrase and the duress key are optional.

Retrieval

When retrieving an encrypted note, to be fully RESTful, if a passphrase was used to decrypt the note, then that will need to be passed as part of the URL. I'm thinking something like this:

https://example.com/api/[random_url]?p=Y6stJWMfaUT5E3L4KeSetezQ

This will return only the plaintext of the note, and no underlying HTML from any templates.

notes should not reveal any internal intormation while at rest on the server. Currently they show at least

1. Whether a passphrase has been set
2. Whether a duress key has been set
3. The size of the note ( modulo AES block size )

I've created a new branch that resolves issue 1.

If a note has a passphrase then the HMAC tag can not be calculated from the URL so we do not need a '.key' file. This simplifies fetch_url also.

This would be compatible but I've removed zlib. I'm not happy with the zlib try catch block on decryption. If the HMAC tag matches we've got what we encrypted and should return data NOT do a second check ?

To solve 2. I'm not sure. Any ideas

To solve 3. I'd pad all notes to 100KB in size. If notes that have more data than they would be split and chained together to reconstruct.

I ran into two problems while trying to install d-note today, and neither was straightforward to diagnose.

One problem is that only two explicitly hard-coded locations are searched for d-note.ini. The first path searched is /etc/dnote, the other is ~/.dnote ... but the instructions do not explicitly state that only these paths are supported. They do say that one of these paths is recommended ... but unless the code is modified, that should say required. I think that the current directory as well as the location of the .wsgi file should be searched, and possibly some other well-known locations in the git working directory. The generate_dnote_hashes script seems to use the same code.

The other problem is the permissions on the dconfig.py file, which may be the problem that the user who filed issue #48 was having. This problem took even longer to track down than the limited ini file locations, because I had to add debug lines to the script that would give a better stacktrace. Only after adding some lines for debugging was I able to determine that dconfig couldn't be imported, despite its location being added to sys.path.

Here's the cause of the second problem: I ran "generate_dnote_hashes" as root. The dconfig.py file was created with 440 permissions, owned by root:root. This meant that the apache user was unable to read the file, so dconfig couldn't be imported, and the traceback in the apache error log was useless except for pointing me at line 47 of note.py. The printed note about needing to run generate_dnote_hashes was NOT included in the apache error log, so I never saw that output.

For both problems, the message for the traceback must be improved so the user has some idea of what's actually gone wrong. For the second problem, the permissions on dconfig.py should be at least 444, if not 644. I can work up a patch for the traceback messages, but I haven't investigated the permission issue yet to see if it's something I know how to fix.

If I follow the install-info to the letter, it fails due to non-existent/non-readable data-dir. I needed to manually do the following for it to start working:

mkdir /var/www/dnote/dnote/data
chown www-data:www-data /var/www/dnote/dnote/data
service apache2 restart

Whether this needs to be added as a manual step in the install docs or if the app should auto-create a missing dir is a design-choice...

Hi guys,

I followed the installation steps on my Debian server but was not able to have d-node running. I got the infamous 500 Internal Server Error.

I have even tried to give full access to the dnote folder (chown -R www-data:www-data) and even chmod 777, but no success.

Actually, I don't even have a hashcash.db .... At which step it is created?
Any other idea on this issue?

Thank you!

The note_destroy function needs to break when it has destroyed the note, to prevent the while loop from continuously executing.

The note should POST to a generic URL rather than the unique URL. Currently, the index.html has the destination URL in the source. Although highly unprobable, and for all practical purposes, it is impossible, an attack could theoretically be achieved where the page is refreshed until the targeted URL is generated.

If the recipient provides the incorrect passphrase for decrypting the note, the following ZLIB error is displayed. Instead, the application should infinitely ask for the correct passphrase.

  File "/usr/share/pyshared/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/share/pyshared/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/share/pyshared/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/pyshared/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/share/pyshared/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/share/pyshared/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/pyshared/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/share/pyshared/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/aaron/src/d-note/d-note.py", line 145, in fetch_url
    plaintext = note_decrypt(privkey, random_url)
  File "/home/aaron/src/d-note/d-note.py", line 90, in note_decrypt
    return zlib.decompress(unpad(plaintext))
error: Error -3 while decompressing data: unknown compression method

See the various queue projects at http://queues.io/.

I am a little confused by the installation instructions for NGINX. At this point all I see are connection refused messages.

First question in regards to, "Create a uwsgi.ini file in the directory with the application"
Does "The Directory with the Application" mean where you downloaded the repo? Or some other directory?

Second in regards to, "And add the following to that file (you can tweak these settings as required)"
Does this mean I should be changing the path? "/python/path/site-packages/dnote-1.0.1-py2.7.egg/dnote"

Third in regards to "add the following to your sites config (again, you can tweak thsi as needed):"
Which site config should I be modifying? The one located in "/etc/nginx/sites-available/default"

Finally, here is what my access.log is stating
"GET /dnote HTTP/1.1" 502 181 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0"
127.0.0.1 - - [23/Aug/2017:14:32:50 -0500] "\x1Ea\x02\x00\x0C\x00QUERY_STRING\x00\x00\x0E\x00REQUEST_METHOD\x03\x00GET\x0C\x00CONTENT_TYPE\x00\x00\x0E\x00CONTENT_LENGTH\x00\x00\x0B\x00REQUEST_URI\x06\x00/dnote\x09\x00PATH_INFO\x07\x00/dnote/" 400 181 "-" "-"

and here is my error.log:
[error] 3538#0: *1 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "GET /dnote HTTP/1.1", upstream: "uwsgi://127.0.0.1:8081", host: "127.0.0.1:8081"

Here is my sites-available default config:

  server {
        listen 8081;
        listen 127.0.0.1:8081;

        root /usr/share/nginx/html;
        index index.html index.htm;

        # Make site accessible from http://localhost/
        server_name localhost;

        location = /dnote { rewrite ^ /dnote/; }
        location /dnote/ { try_files $uri @dnote; }
        location @dnote {
                include uwsgi_params;
                uwsgi_param SCRIPT_NAME /dnote;
                uwsgi_modifier1 30;
                uwsgi_pass 127.0.0.1:8081;
        }

}

When decrypting the note, it is possible that the recipient inputs the passphrase incorrectly, thus throwing the following ZLIB error. The recipient should be presented with providing the passphrase again, making infinite attempts at decrypting the note.

  File "/usr/share/pyshared/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/share/pyshared/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/share/pyshared/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/pyshared/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/share/pyshared/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/share/pyshared/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/pyshared/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/share/pyshared/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/aaron/src/d-note/d-note.py", line 145, in fetch_url
    plaintext = note_decrypt(privkey, random_url)
  File "/home/aaron/src/d-note/d-note.py", line 90, in note_decrypt
    return zlib.decompress(unpad(plaintext))
error: Error -3 while decompressing data: unknown compression method

Rather than providing the FQDN and URI, assume that the service is known, and provide the ability to just copy the URI, with a form text field on decrypting a given URI.

A cron-like system needs to be in place for securely purging expired, unopened notes. Keeping it platform-agnostic, this should probably run inside Python, rather than relying on the operating system cron.

PyCrypto 2.7 introduces some authenticated block cipher modes. They can be found at https://github.com/dlitz/pycrypto/blob/af058ee6f5da391a05275470ab4a4a96aa22b350/ChangeLog. The authenticated modes are:

• CCM
• EAX
• GCM
• SIV

While using HMAC-SHA512 is fine, it would be nice to switch to one of these modes by default, and have the capability of falling back to CBC for old encrypted notes. GCM is probably the preferred choice of the four, due to its parallel processing capabilities, good performance. EAX could be a good alternate.

If moving to an authenticated block cipher mode is not feasible, for whatever reason, when SHA3 becomes standardized, and if it is introduced into PyCrypto, I would like to switch to it taking advantage of the sponge function, rather than relying on SHA2, even though there have not been any strong security weaknesses of SHA2.