NewsLynx Core is an expandable open-source toolkit for building modular content analytics workflows. It provides a fully RESTful API as well as a comprehensive python client and command line interface.

NewsLynx Core was built to power newslynx-app but is capable of powering a diverse range of potential applications, as well, including:

• A Mention.net-like pipeline for your personal or company blog.
• A Flexible timeseries store for content metrics which will automatically summarize and compare your data, as well as enable the additional of custom, computed metrics.
• A framework for configuring, scheduling, and monitoring arbitrary python jobs via API.
• A content-extraction API.

For most applications, refer to our installation guide. If you'd like to setup a development environment, following the instructions below for MacOS X. If you'd like to spin up a Virtual Machine, check out our automation guide.

NOTE We recommend using Postgres APP. However, if you prefer the brew distribution, make sure to install it with plpythonu.

$ brew install postgresql --build-from-source --with-python

(Re)create a postgresql database

# If you already have a database called `newslynx`, delete it first
$ dropdb newslynx 
$ createdb newslynx

Install redis:

$ brew install redis

Open another tab in your shell and run:

$ redis-server

NOTE we recommend that you install newslynx in a virtual environment.

First clone this repository and move into it's root directory

$ git clone https://github.com/newslynx/newslynx-core.git
$ cd newslynx-core

OPTIONAL Set your configuration. If you don't do this, we will fallback on the app's default configuration file.

Run this command when working with newslynx-app

$ make app_install

[Expert mode] don't install the app's default sous chefs, tags, or recipes.

$ make bare_install 

• In debug mode: newslynx debug
• Debug mode with errors: newslynx debug --raise-errors
• Production gunicorn server: bin/run

$ bin/start_workers

$ bin/stop_workers

$ newslynx cron

newslynx is a complicated applcation, to simplify testing...

This repository contains a Makefile which provides access to the following commands:

• make clean

  • clean out cruft generated by pip.

• make clean_sc

  • clean out cruft generated by pip in ~/.newslynx/sous-chefs

• make install

  • installs newslynx
  • will first uninstall newslynx if a version already exists.

• make app_install

  • Installs newslynx, initializes the Super User, and adds the Recipes, Tags, and Sous Chefs needed to run newslynx-app

• make bare_install

  • Installs newslynx and initializes the Super User

• make test_install

  • removes all files under ~/.newslynx
  • runs make app_install
  • runs pip install -r test-requirements.txt

• distribute

  • creates a new version of newslynx on PyPI

• Configuration
• First steps
• Understanding Sous Chefs
• Understanding Recipes
• Understanding Metrics
• Understanding Tags
• Understanding Content
• Understanding Events
• Full API Reference
• Command Line Interface
• Write your own Sous Chef

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.