foxmask / django-th Goto Github PK

View Code? Open in Web Editor NEW

1.3K 1.3K 127.0 4.26 MB

:snake: Trigger Happy - The bus :bus: for your internet services

Home Page: https://foxmask.org/tag/triggerhappy.html

License: BSD 3-Clause "New" or "Revised" License

Python 67.49% CSS 0.28% HTML 32.03% JavaScript 0.14% Dockerfile 0.06%

automation django microservices privacy python-3-6 self-hosted trigger-happy

django-th's Introduction

Trigger Happy

Automate the exchanges of the data between the applications and services you use on the web.

Make Twitter talk to Mastodon, make Github talk to Mattermost, store your favorite tweets by creating notes in Evernote, follow RSS feeds and post each news in Wallabag, Pocket or Evernote.

The possibilities are too numerous to name all of them, but with that project you won't have to raise your little finger at all: automate everything and make your life easier.

And last but not least, as this is your project, all the credentials you used to give to IFTTT and consorts, are now safe in your hands.

Requirements

The minimum requirements are the following:

Installation

pip install django-th

Documentation

For installation and settings, see http://trigger-happy.readthedocs.org/

Archiving the projet (5/1/2020)

I could try to migrate to django 3.0, but I don't use any of the supported services anymore, so I can't migrate to django 3.x

The project use a lots of tricks to handle Form Tools Wizard, required to create triggers ;) And many many others ones to load service class automatically, use webhooks and so on.

The Furture is now in yeoboseyo, a little "Trigger Happy" made with starlette (and its projects ecosystem)

django-th's People

Contributors

Stargazers

Watchers

Forkers

zenweasel sainathnaga pombreda philippeowagner rajsid1 pombredanne christoschristofidis pouete sidaga averroes 0wnrepo escalation mikebronfman workwebresources thunderflurry cloudxtreme reidharry mrphishxxx bjzt1016 rookieokky cyber-forensic dgo-list anishihara guailong618 maniacs-oss viktor-evdokimov tirasu lynxmac qianxueseng-com cganey dgreyling codeinpeace himanshpal jeffcjohnson ideaplexus exexzian octogene thearchiver awesome-python techscientist bharathi26 cerya jun9 cgiogkarakis yonatanlebo svpn testmana2 jakop345 jayjay1 datahack-ru ufgstores ldanilov billthebest mantisware ciphersson chrisholter bobquest33 learnerwu prsahu nagyistge julianpistorius cgnkev nstallbaumer m8e vaibhavsingh97 denvaar h-chauhan guptarohit koalie alabhyavaibhav scomert alonisser pal0r fkztw aping1 itsnikka ehsan-keshavarzian timonweb-forks adriancarayol ayukha kfdm purp parth-choudhary ykankaya thaveedu farmers-tan firuza rubbish822 scientificpotato dahburj hmphu su57 alirizakeles forksbot sanketrathi sekmet boomdrk barrygolden mlicam hammsidh

django-th's Issues

Editing triggers

this should permit to change the URL of the provider

Details in the body

add the source of the data in the body like :

"from foobar.com" foloowing by the link of the page

Make a Generic View to handle 2 models at once

Now that the Wizard is working great, I have to drop the FBV I did to handle RSS+Evernote and make a generic (FC)BV that will generate a generic form from the 2 models we plug each other during the FormWizard.

Fire.py : generic call of datas

From the service provider calls line 41, we have to get data from "content", thus line 46 we could read data in the for loop like this :

 content = data.content.value or data.content

instead of the actual

 content = data.content[0].value

which is very "RSS" oriented ;)

This will permit to integrate many other services (provider) more easyly.

Encoding issue with pytidy

when creating a note to evernote the sanitize function which call pytidy ; return a valid XML but drop ALL the UTF-8/non-ascii chars ...

Need to make some test with LXML

OPML handling

Add the feature that permits to load a OPML file like http://planet.python.org/opml.xml and create one trigger for each.
the wizard will be :

OPML file to load
Target service to choose and folder/tag to select
once it's done the description will contain:
the text property of the OPML source + "Rss to " + choosen Service name
Thus it will be easier to distinguish each of the Feeds in the list.

number of active triggers

on top of the main page ; display the total of triggers created and number of activated one
useful when we start to have more than one page to turn ;)

Python 3.4.0 Compatibility

Search engine

to filter by anything we want, for example by target service, by provider service, by triggered date etc -

Have to see from here http://django-haystack.readthedocs.org

Monitoring Provider/Consumer of the data sent

We could keep the log of the last event that has been triggered and stored to a service

For example this :

"From ProviderServiceA: string of the event - To ConsumerServiceB"

Could be view from a new Provider/Consumer page that will list the last triggered data

lets i18n the existing source

https://docs.djangoproject.com/en/1.5/topics/i18n/translation/

list of services created by users

to be done on the admin panel

Description of the Activated Service

For each service, display :

name,
description (what does that service)
picture/logo

this will help the user to know what is the goal of the service he can activate

Dont update when Exception occurs

When a service raises an exception, return False then:
so

                            consumer(
                                service.consumer.token, service.id, **data)

become

                            status = consumer(
                                service.consumer.token, service.id, **data)

if status if false
also initialse

status = False

after

to_update = False

and

            if to_update:
                logger.info("user: {} - provider: {} - consumer: {} - {} = {} new data".format(
                    service.user, service.provider.name.name, service.consumer.name.name, service.description, count_new_data))
                update_trigger(service)
            else:
                logger.info("user: {} - provider: {} - consumer: {} - {} nothing new".format(
                    service.user, service.provider.name.name, service.consumer.name.name, service.description))

will become

            if to_update:
                logger.info("user: {} - provider: {} - consumer: {} - {} = {} new data".format(
                    service.user, service.provider.name.name, service.consumer.name.name, service.description, count_new_data))
                update_trigger(service)
            else:
               if status:
                logger.info("user: {} - provider: {} - consumer: {} - {} nothing new".format(
                    service.user, service.provider.name.name, service.consumer.name.name, service.description))
             else: 
                logger.info("user: {} - provider: {} - consumer: {} - {} AN ERROR OCCURS ".format(
                    service.user, service.provider.name.name, service.consumer.name.name, service.description))

Trigger List is h+2

the trigger list displays the last triggered date but is h+2

Renew token

Actually, when a token is expired, fire.py exit with an exception.

We have to check the token in a try/catch to avoid to exit.

Also we need to show on the service page, that the service is disable because of the expired token

Wallabag integration

from here and there both project are linked. \o/ Have to finished what was started

validate the token again

if, from the external service, we drop the authorization we grant to Trigger Happy, then Trigger Happy fails to handle the job.

We Need to add a button to renew the token.

Django CMS integration

I put here a peace of code as a starting point :

cms_app.py to put in at the same level of models folder

# -*- coding: utf-8 -*-

from django.utils.translation import ugettext_lazy as _

from cms.app_base import CMSApp
from cms.apphook_pool import apphook_pool


class TriggerhappyApp(CMSApp):
    name = _("Trigger Happy")        # give your app a name, this is required
    urls = ["django_th.urls"]       # link your app to url configuration(s)
    app_name = "django_th"          # this is the application namespace

apphook_pool.register(TriggerhappyApp)  # register your app

cms_plugins to put at the same level of models folder:

# -*- coding: utf-8 -*-
from cms.plugin_base import CMSPluginBase
from cms.plugin_pool import plugin_pool
from django_th.models import TriggerHappyPlugin
from django.utils.translation import ugettext as _


class CMSTriggerHappyPlugin(CMSPluginBase):
    model = TriggerHappyPlugin  # Model where data about this plugin is saved
    name = _("Trigger Happy Plugin")  # Name of the plugin
    # template to render the plugin with
    render_template = "cms_plugin.html"

    def render(self, context, instance, placeholder):
        context.update({'instance': instance})
        return context

# register the plugin
plugin_pool.register_plugin(CMSTriggerHappyPlugin)

Template to create:

{% extends "base.html" %}
{% load static %}
{% load url from future %}
{% load i18n %}
{% load django_th_extras %}

{% block title %}{% if user.is_authenticated %}{% trans "My Triggers" %} - {{ user.username }} - {{ current_site.name }}{% else %}{% trans "Home" %} - {{ current_site.name }}{% endif %}{% endblock %}


{% block content %}

    <!-- Fixed navbar -->
    <div class="navbar navbar-default navbar-fixed-top" role="navigation">
        <div class="container">
            <div class="navbar-header">
                <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
                    <span class="sr-only">Toggle navigation</span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                </button>
                <a class="navbar-brand" href="{% url 'base' %}" title="{% trans "Home" %}">Trigger Happy</a>
            </div>
            <div class="navbar-collapse collapse">            
                {% if user.is_authenticated %}
                <ul class="nav navbar-nav">
                    <li><a href="{% url 'base' %}" title="{% trans "Home" %}"><span class="glyphicon glyphicon-home"></span></a></li>
                    <li>{% if nb_services > 1 %}<a href="{% url 'create_service' %}" title="{% trans "Create a new trigger" %}">{% else %}<a href="#" title="You can't create a new trigger until 2 services are activated">{% endif %}<span class="glyphicon glyphicon-plus"></span></a></li>
                    <li><a href="{% url 'user_services' %}" title="{% trans "List of your own activated services" %}"><span class="glyphicon glyphicon-tasks"></span></a></li>
                    {% block filter_trigger %}
                    {% include "filter.html" with trigger_filter_by=trigger_filter_by %}
                    {% endblock %}
                    {% url 'profiles_profile_detail' request.user.username as profiles_profile_detail %}
                    <li><a href="{% url 'logout' %}" title="{% trans "log out" %}"><span class="glyphicon glyphicon-off"></span></a></li>
                </ul>
                {% endif %}
            </div><!--/.nav-collapse -->
        </div>
    </div>    

{% if user.is_authenticated %}
        <div class="col-md-12">
            <div class="col-md-5">
                <h2>{% trans "My Triggers" %}</h2>
            </div>
            <div class="col-md-6 trigger-summary">
                {% trans "Summary" %} :
                <span class="label label-info">{{ nb_services }}</span> {%  trans "activated" %} <span class="label label-default">{% trans "services" %}</span>
                <span class="label label-info">{{ nb_triggers.enabled }}</span> {%  trans "enabled" %} <span class="label label-default">{% trans "triggers" %}</span>
                <span class="label label-warning">{{ nb_triggers.disabled }}</span> {%  trans "disabled" %} <span class="label label-default">{% trans "triggers" %}</span>
                <hr/>
                {% trans "Actions" %} : <a href="{% url 'trigger_switch_all_to' 'off' %}" title="{% trans "this action will set all the triggers off"%}">{% trans "Mark all Off" %}</a> - <a href="{% url 'trigger_switch_all_to' 'on' %}" title="{% trans "this action will set all the triggers on"%}">{% trans "Mark all On" %}</a>
            </div>
        </div>
    {% if triggers_list %}
        {% if is_paginated %}
        <div class="col-md-12 pagination">
            <ul>
                {% if page_obj.has_previous %}
                    <li><a href="{% url 'home' %}?page={{ page_obj.previous_page_number }}">{% trans "previous" %}</a></li>
                {% endif %}
                    <li class="active"><a >
                    {% blocktrans with page_number=page_obj.number total_of_pages=page_obj.paginator.num_pages %}
                    Page {{ page_number }} of {{ total_of_pages }}
                    {% endblocktrans %}</a>
                    </li>
                {% if page_obj.has_next %}
                    <li><a href="{% url 'home' %}?page={{ page_obj.next_page_number }}">{% trans "next" %}</a></li>
                {% endif %}
            </ul>
        </div>
        {% endif %}
        {% for trigger in triggers_list %}
        <article>
        <div class="trigger-record {% if trigger.status %}trigger-enable{% else %}trigger-disable{% endif %} col-md-12">
            <div class="col-md-7 trigger-text">
                <a href="{% url 'edit_provider' trigger.id %}" title="{% trans 'Edit' %} {{ trigger.provider.name|service_readable|lower }}" >{{ trigger.provider.name|service_readable }}</a> <span class="glyphicon glyphicon-chevron-right"></span> <a href="{% url 'edit_consumer' trigger.id %}" title="{% trans 'Edit' %} {{ trigger.consumer.name|service_readable|lower }}" >{{ trigger.consumer.name|service_readable }}</a>
                <h3><a href="{% url 'edit_trigger' trigger.id %}" title="{% trans 'Edit the description' %} ">{{ trigger.description|safe|escape }}</a></h3>
            </div>
            <div class="col-md-5">
                {% if trigger.status %}
                <a class="btn btn-lg btn-info" role="button" href="{% url 'edit_provider' trigger.id %}" title="{% trans 'Edit your service' %} {{ trigger.provider.name|service_readable|lower }}" ><span class="glyphicon glyphicon-pencil icon-white"></span> {{ trigger.provider.name|service_readable|lower }}</a>

                <a class="btn btn-lg btn-info" role="button" href="{% url 'edit_consumer' trigger.id %}" title="{% trans 'Edit your service' %} {{ trigger.consumer.name|service_readable|lower }}" ><span class="glyphicon glyphicon-pencil icon-white"></span> {{ trigger.consumer.name|service_readable|lower }}</a>

                <a class="btn btn-primary btn-lg" role="button" href="{% url 'trigger_on_off' trigger.id %}" title="{% trans 'Set this trigger off' %}"><span class="glyphicon glyphicon-off icon-white"></span></a>

                {% else %}
                <a class="btn btn-lg btn-info disabled" role="button" href="#" title="{% trans 'service disabled' %} {{ trigger.provider.name|service_readable|lower }}" ><span class="glyphicon glyphicon-pencil icon-white"></span> {{ trigger.provider.name|service_readable|lower }}</a>

                <a class="btn btn-lg btn-info disabled" role="button" href="#" title="{% trans 'service disabled' %} {{ trigger.consumer.name|service_readable|lower }}" ><span class="glyphicon glyphicon-pencil icon-white"></span> {{ trigger.consumer.name|service_readable|lower }}</a>

                <a class="btn btn-success btn-lg" role="button" href="{% url 'trigger_on_off' trigger.id %}" title="{% trans 'Set this trigger on' %}"><span class="glyphicon glyphicon-off icon-white"></span></a>

                {% endif %}
                <a class="btn btn-lg btn-danger" role="button" href="{% url 'delete_trigger' trigger.id %}" title="{% trans "Delete this trigger ?" %}"><span class="glyphicon glyphicon-trash icon-white"></span></a><br/>

            </div>
            <footer>
            <div class="col-md-12">
                    <p><span class="glyphicon glyphicon-calendar"></span>{% trans "Created" %} {{ trigger.date_created }} - 
                    <span class="glyphicon glyphicon-calendar"></span>{% trans "Triggered" %} {% if trigger.date_triggered %}{{ trigger.date_triggered }}{% else %} {% trans "Never triggered" %}{% endif %}</p>
            </div>
            </footer>
        </div>
        </article>
        {% endfor %}
        {% if is_paginated %}
        <div class="col-md-12 pagination">
            <ul>
                {% if page_obj.has_previous %}
                    <li><a href="{% url 'home' %}?page={{ page_obj.previous_page_number }}">{% trans "previous" %}</a></li>
                {% endif %}
                    <li class="active"><a >
                    {% blocktrans with page_number=page_obj.number total_of_pages=page_obj.paginator.num_pages %}
                    Page {{ page_number }} of {{ total_of_pages }}
                    {% endblocktrans %}</a>
                    </li>
                {% if page_obj.has_next %}
                    <li><a href="{% url 'home' %}?page={{ page_obj.next_page_number }}">{% trans "next" %}</a></li>
                {% endif %}
            </ul>
        </div>
        {% endif %}
    {% else %}
        <div class="trigger-record col-md-12">
            <div class="alert alert-info">
                <button type="button" class="close" data-dismiss="alert">&times;</button>
                {% trans 'No trigger yet' %}
            </div>
            {% if nb_services < 2 %}
            <div class="alert alert-danger">{% trans "Before creating a trigger" %} <span class="label label-danger">{% trans "you have to activate" %}</span> <a href="{% url 'add_service' %}">{% trans "at least 2 services" %}</a>. </div>
            {% else %}
            <p>{% trans "You can now proceed by creating" %}<a href="{% url 'create_service' %}"> {% trans "your first trigger" %}</a></p>
            {% endif %}
        </div>
    {% endif %}
{% else %}
            <div class="jumbotron">
                <p>
                    <strong>Trigger Happy</strong> is up !
                </p>
                <p>Have Fun</p>
                <a class="btn btn-primary" href="{% url 'login' %}">{% trans "Log in" %}</a>        
            </div>
{% endif %}
{% endblock %}

Poll : waiting for result :)

next dev will be from http://www.foxmask.info/post/2013/07/13/sondage-quels-services-utilisez-vous-le-plus/

en/disable all triggers

add 2 links to be able to disable / enable all the triggers in a row

check the title

when grabbing source data, check if a title is provided to avoid something like this:

Traceback (most recent call last):
  File "./fire.py", line 123, in <module>
    main()
  File "./fire.py", line 119, in main
    go()
  File "./fire.py", line 72, in go
    service.consummer.token, title, content, service.id, extra)
  File ".../django_th/services/my_evernote.py", line 120, in save_data
    created_note = note_store.createNote(note)
  File ".../local/lib/python2.7/site-packages/evernote/api/client.py", line 138, in delegate_method
    )(**dict(zip(arg_names, args)))
  File "..../local/lib/python2.7/site-packages/evernote/edam/notestore/NoteStore.py", line 4793, in createNote
    return self.recv_createNote()
  File "..../local/lib/python2.7/site-packages/evernote/edam/notestore/NoteStore.py", line 4817, in recv_createNote
    raise result.userException
evernote.edam.error.ttypes.EDAMUserException: EDAMUserException(errorCode=2, parameter='Note.title')

if the title is empty, do not handling this data and continue to the next record

Instagram module

Creating a New Module in Trigger Happy

instagram API to use

see feed from http://instagram.com/developer/endpoints/users/

Twitter support

start it here https://github.com/foxmask/django-th-twitter

Paginator

add a paginator to the list of Triggers

perf improvement

split __init__.py from models and forms in a base.py module

thus this will avoid to load the content on each __init__.py with things that are not expected.

Log : number of new threads & name of the user

when fire.py run, we should add the user at first then the service provider ; consummer ; number of new data ; then the description of the trigger

smth like :

[user] [provider] [consummer] [num_of_new_datas] [description]

thus :
we'll be able to track the stability of the service for each user.

TriggerAction

When 2 services are choosen, create an action.
eg.

From a RSS Feeds Create a tweet from the RSS title,
From a RSS Feeds Create a note evernote from the RSS content

Provider data without date

When a provider does not have a date of creation (or something like that), trigger-happy creates the same data to the consummer part, at each launch of fire.py ...

eg : RSS feeds does that.

But other provider does not have a date and are able to get fresh datas between two launches.

eg : pocket does that

So several possibilities :

a solution could be to ignore those data without a (creation) date, but in that case we wont be able to handle potentially a lot of service :/
another solution could be to set a default date to midnight.
another solution could be to store the grabbed datas in a cache which, this one, will have a date and keep this cache during a given period. After this period, the data-without-a-date will no longuer exist on the Provider part.
another solution could be to set the date to the date of the previous data we grabbed ; eg get the previous date of the feed, and if it's the first one, set it to midnight ...

create a add_service command to start to support a new service

something like

python add_service.py foobar

will have to add :

a class my_foobar in the services directory which ineherits of services class
a class foobar in the models directory
a class foobar in the form directory
a template for the wizard of the service in templates/foobar/wiz-foobar.html
add "from .models.foobar import Foobar" in views.py
add "from .foobar import Foobar" in models/init.py
add in settings.py to the tuple TH_SERVICES :

     'django_th.services.my_foobar.ServiceFoobar',

add in settings.py a new tuple :

TH_FOOBAR = (

    'consumer_key': 'abcdefghijklmnopqrstuvwxyz',
    'consumer_secret': 'abcdefghijklmnopqrstuvwxyz',
)

and probably something with the TH_WIZARD_TPL tuple too.

Check the validity of the RSS feed

use this http://pythonhosted.org/feedparser/bozo.html to check if the feeds are ok then displays a warn to the user if the feeds is not well formed

deal with ajax request when disabling/enabling triggers

currently when we enable / disable a trigger on page 2,3,4 etc... we go back to page 1.
Manage the action with ajax then we'll stay of the page we are.

fire.py out of order with PostgreSQL 9.1

With multiprocessing this gives those error https://gist.github.com/foxmask/9113543

The script works in single process or with multiprocess but with SQLite.

Same for MySQL 5.5 - multiprocessing does not work.

setup.py to deal wih html template

when packaging to pypi html are not provided

published_parsed not found

ATOM feeds do not always have updated/issued node thus fire.py failed and crashed

  File "./fire.py", line 63, in go
    to_datetime(data.published_parsed) >= service.date_triggered:
  File "../lib/python2.7/site-packages/feedparser.py", line 417, in __getattr__

We need to check if published_parsed exists before using it otherwise try to find another date from : http://pythonhosted.org/feedparser/date-parsing.html

by service name
by status (enable/disable)

Twitter Support

use https://pypi.python.org/pypi/twitter
this will need to make a dedicated module django-th-twitter

this should be used to :

post a tweet from a source of data for example RSS
get new tweet to write datas in another service for example add a note in Evernote