siftlogic / erl-dns Goto Github PK

Add support for synchronizing records via RabbitMQ

Each erl-dns node should subscribe to a fan-out queue to receive changes

Support a question-response set of queues for each node so that the node can request a replay of changes from a specific hash point

A huge issue with the geolocation service brings this problem. Feature and test definitions will be created as issue #6 is being developed.

** Also, region data stored in the DB needs to be transferred to slave as well so they can also create their own look up table that is up to date with master.

This module needs some cleanin! Best to make it as clear as possible, attempt to remove redundant code and make everything follow a clear code path.

All servers should have a default region database that should be built at startup.

Need to fix it..

For example:

I have a wildcard georecord for "us-east" under the name _geo.us-east.*.example.com.
I get a query for wookies.example.com, it should be able to match the geo record with that.

FEATURE

• change ips in config to be a list
• add functionality for new instances to be started based on how many IPs are in the list.
• check for 0.0.0.0 address in the list before adding any other IPs.
• event_handler should add the children to the nodes started.
• worker pools should be in place for every server.

TESTING

• Test for different IP representations be inserted in config

Need to make sure any time a zone is deleted, we also remove it from the orddict. In another thought zone addition must also result in the orddict being updated.

The Master server should be able to be 'hidden' or 'public'.

'hidden' means it does not handle any requests other than 'AXFR' from Slave servers.

'public' means it handles requests for which the server is listed as a valid 'NS' and contains the zone file

FEATURES

• Master for a zone configured to run as 'hidden' or 'public'
• Master only handles AXFR requests from authorized slaves when 'hidden'
• Master handles all other requests when 'public'

TESTING

• Master does not respond to any requests other than AXFR from authorized slaves when hidden
• when 'public', handles requests for which the server is listed as a valid 'NS' and contains the zone file

Zones should be able to support geo-located dns records.

IE, I should be able to specify multiple records for a single 'value' based on the location of the query source.

EG
{<<"example.com">>, <<"A">>, {192,168,1,2}, <<"US">>}, %% record served is query source is US ip
{<<"example.com">>, <<"A">>, {192,168,1,2}, <<"CA">>}, %% record served is query source is CA ip
{<<"example.com">>, <<"A">>, {192,168,1,2}, <<"GB">>}, %% record served is query source is GB ip
{<<"example.com">>, <<"A">>, {192,168,1,2}, undefined} %% record served is query source is ip from any other location

Requirements:

• include egeoip as a dependency
• include {support_geo, true | false}, with default false as a per-server config option
• when support_geo is true, determine country for each query and use country in record lookup

We may want to investigate whether or not splitting zones and records into separate tables is a good idea as the match patterns in mnesia may be slow/complex when having to dig into the existing 'zones' only based storage.

By splitting it would also be possible to have entire 'geo-located' zones instead of just records (maybe bad idea as introduces complexity)

FEATURES

• Region data should not have duplicates
• Reference geolocation data in a dns_rr record
• Backward compatiblity must be maintained when support_geo is false.
• Geo record must be represented in a dns_rr record type.
• erldns must query geoip db for record location and match the region with the geo record
• geoip records should fall under geographic locations (US-East, US-Central, US-West, etc).
• Best match record should be found by comparing location of query to location of response, in otherwords it must match the client IP location to the geo record (if any) and return it.
• -- If no match is found, use the default record

TEST

• Ensure geo located regions are returning their records.

FEATURE

• mnesia should be backed up every interval specified in "core"
• if a node goes down, "core" needs to reload that node with the current version of the DB
• "core" should have backups for every interval, and be able to restore from those points
• backup directories should have a configurable variable in the config.

TESTING

• common test for backup functionality
• common test for restore functionality
• common test for node crash/restore functionality

We want a single server to be able to run as the DNS MASTER.

AXFR and NOTIFY should only be supported in TCP based queries

The Master will additionally send NOTIFY messages to Slave servers whenever a Zone changes. The Slave servers would only be those servers listed as slaves for that specific zone.

This would require:

• server to read from a zone in order to check if it is master or slave. If it is master, it should be responsible for sending all NOTIFY messages when a zone changes.
• Supporting NOTIFY messages in erldns_tcp_server
• verifying correct handling of AXFR in erldns_tcp_server
• Having slave servers process the NOTIFY messages, check the Serial Number against what they have stored for the zone, and if necessary, initiating a AXFR request to the MASTER server, receiving the response from the MASTER, and then updating the stored zone (and flushing any cached records)

FEATURES

• add record (increment serial)
• delete record (increment serial)
• update record (increment serial)
• single server to be able to run as the DNS MASTER.
• other servers run as slave
• FIX zone transfer requests
  NOTIFY / AXFR
• AXFR and NOTIFY should only be supported in TCP based queries
• Create a NOTIFY message IAW the RFC.
• -- Master Send NOTIFY on zone change, or on interval
• ---- Slaves should be able to process NOTIFY messages
• ------ Slaves should receive a NOTIFY message
• ------ Verify NOTIFY is allowed to be handled by the sender master on recv NOTIFY
• ------ Verify slave for zone on recv NOTIFY
• ------ Slaves should check the serial number for a zone, and send a AXFR request if there is a mismatch
• -- Slave Send AXFR request
• -- Master Send AXFR
• ---- Verify AFXR request is allowed for the requester's IP on send AXFR
• ---- Verify server is a master for that zone on send AXFR
• ----- Slaves should receive and process AXFR response, flushing old cached records
• Slave should request a zone transfer when a zone is expired
• Master should notify slave when a zone is added
• Slave must add a zone when the master does
• Master should notify slave when a zone is deleted
• Slave must delete record if master has deleted a zone
• Add security to admin module
• '--' denotes a sub task for the task mentioned prior to it.

TESTING

• Server runs as master or slave
• -- Master sends NOTIFY on zone change and handles AXFRs
• -- Slave sends AXFR request when a zone expirers
• Master sends NOTIFY when a zone changes
• slaves process NOTIFY messages, AXFR request/process
• tests for zone record add/modify/delete
• Slave deletes/adds zone when a master does.

This will simply require creation of mnesia schema if it is not already there

EDIT:
Seems to be a bit deeper of a problem.

To recreate:

1. Load erl-dns with storage type mnesia and a clean DB
2. erldns_zone_cache:zone_names() should return empty
3. Do these commands in shell:

• rr(erldns_zone_cache).
• Zone = #zone{name = <<"test">>}.
• erldns_storage_json:insert(zones, Zone). %%bypassing mnesia to get a crash

4. erldns_manager:setup_zone_expiration_orrdict() crashes where it is calling erldns_zone_cache:zone_names().

Necessary evil.

FEATURES

• Create generic storage api module erldns_storage through which all storage operations will flow. API functions:
  • create_table
  • insert
  • update
  • select
  • delete
  • delete_table
  • backup_table
  • backup_tables
  • empty_table
• Create erldns_storage_json module which will encapsulte JSON + ETS based storage logic
• Create erldns_storage_mnesia module which will encapsulte mnesia based storage logic using disk_copies
• replace all existing ets:Func calls with calls to erldns_storage API module
• replace all ets in zone_cache with API calls
• Support initial loading of JSON configs into mnesia db through API
• Move JSON handlers to API (zone_loader)
• mnesia storage location to be set by a config

TESTING

• verify all works and tests pass with JSON storage module
• create tests for mnesia storage module
• verify all works and tests pass with mnesia storage module
• move all tests to common test

The zones are not being deleted in the orddict. Also the logic when creating the orddict is flawed. It is using the "mounted IP" addresses and using an incorrect IP when testing to see if the node is master for the particular zone. We should pass in the bind ip to the module when starting.

• Zone should be added to orrdict when a new zone is added
• Zone should be deleted to orddict when a zone is deleted
• correct logic for setting up orddict