mitodl / configuration Goto Github PK

See https://github.com/TechTV/TechTV/issues/168 (I don't really want to loop TechTV into our ZenHub boards just for this one small issue)

Depends on #18 ?

Would fix ZenDesk #3140

using the latest version of Chrome, all equations have a little bar next to them

Consider Ansible Vault or Hashicorp vault

What about deployment?

See ZenDesk #3152

The course EnX would like to use an LTI app so students can identify where they are on a Google Map.

The LTI application is available from https://github.com/UQ-UQx/response-map

Discovery Questions:

• Can we host it on Heroku?
• Can we host it on MIT's Heroku account (free for us)
• Can the app support multiple courses, or will we need a course per app?
• Make it work under TLS

If it's helpful, I can probably track down someone from UQx who worked on this.

via Shelly Upton:

Some examples from Delft if you’re enrolled/can enroll
https://courses.edx.org/courses/DelftX/Frame101x/1T2015/16dee7ee1e5b498ab06b19f7f78f87a0/
https://courses.edx.org/courses/DelftX/DDA691x/3T2014/a56ac0c9e39b40cf924b526ec0fe2d17/

reinvest in ELK or use a service?

if ELK, we need a 3-node cluster
or would we use Amazon's Elasticsearch service?

Need to consider FERPA requirements

• Ask Logentries about FERPA compliance
• Ask IS&T about meaning of DLT's claim of FERPA compliance
• Ask Loggly about FERPA compliance

Currently logs-rp doesn't have any data in it, due to running out of disk space on the elastic search server.

Follow-up issue to discovery in mitocw/edx-platform#199

The filebased cache is throwing dozens of 500 errors every day, and we've confirmed that users see some (if not all) of them.

Let's switch the cache to memcached, or possibly redis. memcached should be easier to set up (because that's what all the other caches use) but it may consume a lot of RAM (given the size of some of our courses). Redis might not be too hard to set up, if we use a service.

Talk to customer rep and try to get a volume discount -- hopefully retroactive

Consider Geofront

will have to update all the security rules once this is in place.

The DataDog service supports a large number of integrations, many of which would be useful for our purposes. We should fork the DataDog formula and addd support for integrations that we find useful.

How much would an additional MITx stack cost us?

Scope for now:

1-5 large courses
appx 20k user registrations (mostly using social auth)
unestimated number of simultaneous users

Is this more than partners.mitx.mit.edu? less than lms.mitx.mit.edu?

• edx-platform code for conditionally exporting normalized grades
• mitodl/PyLmod#62 pylmod release
• mitodl/lmod_proxy#51 lmod_proxy release

This issue covers support for the release master

Upgrade all the MITx-devops repos to work with Cypress.

There are related issues on multiple repositories.

This is for tracking the tasks necessary to rearchitect our deployment and management of virtual infrastructure. Rather than our current approach of maintaining long-running instances and patching/upgrading them in place we are going to be rebuilding the images and deploying copies of them. This will increase deployment speed as well as preventing a lot of deploy-time issues by letting us verify the final state of a deployed instance before it is actually put into production.

• infrastructure testing (mitodl/salt-ops#8)
• test data management (i.e. having a copy of production db for testing) (mitodl/salt-ops#15)
• run edx-app ansible with salt (mitodl/salt-ops#13)
  • write salt states for mongo cluster & build AMIs (5)
  • build salt states for rabbitmq cluster & build AMIs (5)
  • storage
• create images with salt (mitodl/salt-ops#9)
• deploying images (spin up, put in place, rotate out old ones) (mitodl/salt-ops#10)
• discovery: how best to customize edX configuration? (mitodl/salt-ops#14)
• service discovery (story exists)
• Determine order of migration for upgrades (mitodl/salt-ops#16)

Currently, we use DNS naming conventions and/or IP ranges by convention to locate servers.

Consul from hashicorp has a way for servers to announce their services to a central key=value store. Probably a requirement for using Hashicorp Vault.

With VMOS gone, we need a new place to deploy sandboxes, i.e. RPC

(not sure this the right repo for this issue)

Now that we have a new kibana cluster for search logs, we need to work out the details of forwarding the logs from the MITx servers into it. Much of this is already done, but there appears to still be some tweaking to do.

• Add fluentd to servers to capture logs & forward to kibana
• pre-process logs so that they are indexed appropriately
• demo new kibana interface for Peter, Ben and anyone else who needs to review MITx logs
• provision more disk space for kibana indexes

To replace Zenoss

We should already have a datadog account.

Setting up monitoring & alerts is a separate task

update endpoint for systemd

Use salt cloud to call ansible play to deploy instance? Terraform? Or resurrect Jenkins + Shipyard solution?

mitodl/xsiftx#29

It would be good to assess our current monitoring solution (Zenoss). Zenoss does SNMP walks and executes plugins that perform HTTP requests to healthcheck/status URLs to determine availability. It checks what services are running via patterns, etc. It does simple graphing does threshold/pattern based alerting on any data point.

However, we could really benefit from having a couple of things:

• Reactive monitoring: @blarghmatey mentioned this one. There are sometimes things we can expect to happen (on disk, for instance) the require cleanup. For example, Studio imports from git leave a buildup of repositories over time on disk. Instead of trying to build crons for all of these things that periodically check the size of the directory/rotate the oldest repositories, it would be nice to have a service that reacts to inode events, etc - and have these watches be controlled from a central place.
• Maintained support for our integrations: Carson made a Zenoss plugin for Hipchat, but he's gone now and it's not maintained currently, especially since he's not using Zenoss at his new job. While it's probably not going to break anytime soon, It would be nice to use a monitoring service that directly supports our chat combination.
• Not having a crappy, convoluted interface.
• An easy way to automatically add/remove managed machines upon orchestration.

We need to revisit the old Rackspace Private Cloud requirements and plan and update/validate it for VMWare.

This gooogle doc is a starting point, but it assumed a baseline of features that we should reivist.

See ZenDesk #3042

@itsbenweeks is it OK to turn off all mail in the app?

As a user, sometimes I'd like to export an OLX structure that can actually be imported as a course into edX.

This would require an UI for the use to provide the necessary course-level metadata to create a course.

Saltstack instead of Ansible 1.5

• #55 write states for elastic search #55
• #56 write states for fluentd (replacement for logstash)
• #57 write states for a log shipper (maybe just syslog)
• deploy

Build and deploy a Salt master into a new VPC as the starting point for future salt-based infrastructure.

See ZenDesk 3105 All studio users have course creator access.

This will allow for building sandbox AMIs that can be used for quickly creating new sandboxes for testing Micromasters, TeachersPortal, and changes to the edX codebase.

need a way to find, for example, the lastest mongo AMI

This is not the right repo for this issue. But it will have to do for the moment.

This is to replicate techtv database to Ike's quickbase app.

Carson's public key needs to be removed across our inventory.

Write cookie cutter template for saltstack using packer for building images

I'm thinking I'll tag the tasks as "flatfile".

This is a placeholder for https://github.mit.edu/mitx-devops/mitx-ansible/pull/90

• Cut branches for Cypress Hotfix 6
• Mathjax fix
• Remote gradebook enhancement
• maybe mongo_inheritence cache change

See ZenDesk 2995