lcogt / neoexchange Goto Github PK

When testing NEOx we've accidentally sent things to the scheduler. When using the dev server (or even running tests) we should automatically disable submitting to the scheduler.

My code which is intended to read a NEOCP 1-line orbit file from disk and use the Body from the models.py and then tries to call save_and_make_revision() from views.py is dying horribly with the following traceback:

http://dpaste.com/2930VF5

Since the implementation of the site URL prefix, the admin site does not redirect correctly after login. After a successful login, it redirects to:
http://docknode06:8200/admin/login/?next=/admin/
producing a 404 Not Found nginx/1.6.3 error.

It would be helpful, in the overall Targets view (/target/) and Target-specific view (/target/XXXX/) if the prior provisional designation was available for objects which have a new final designation. In the old NEO site, I formatted the cross idents in a single column as 'P10luqgh-->2015 LB2' but this is only a suggestion.

Blocks page is long and in need of either pagination or something more complicated (e.g. pulling in all active blocks which wouldn't be in the first page).

We wish to include those targets that are on the Arecibo target list (http://www.naic.edu/~pradar/) as these 1) may not be on the Goldstone list and 2) DSS-14 will be down for maintenance for 2016 Feb-May.

Unresolved question: do we want a specific origin type for objects that are on both lists (currently common objects will ping-pong from origin=Arecibo to origin=Goldstone and back again depending on which was the last management command).

Today (2015-0-29 21:25 UTC), I tried to submit a block for 2015 SZ2 to CPT during the time the RequestDB had been taken down for upgrades which produced a ServerError (and logged me out so I couldn't see the messages). From /var/www/apps/neoexchange/neox.log:

[29/Sep/2015 21:23:48] ERROR Internal Server Error: /neoexchange/schedule/3161/confirm/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "./core/views.py", line 212, in post
    return super(ScheduleSubmit, self).post(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 215, in post
    return self.form_valid(form)
  File "./core/views.py", line 216, in form_valid
    tracking_num, sched_params = schedule_submit(form.cleaned_data, target)
  File "./core/views.py", line 308, in schedule_submit
    tracking_number, resp_params = submit_block_to_scheduler(body_elements, params)
  File "./astrometrics/sources_subs.py", line 662, in submit_block_to_scheduler
    response_data = client.submit(user_request)
  File "/usr/lib/python2.7/site-packages/reqdb/client.py", line 261, in submit
    response_data = self._submit_to_endpoint(post_req, debug)
  File "/usr/lib/python2.7/site-packages/reqdb/client.py", line 409, in _submit_to_endpoint
    raise ConnectionError(msg)
ConnectionError: Received an error from server: [Errno 111] Connection refused
Your url may be wrong, or the endpoint at 'http://scheduler1.lco.gtn/requestdb/' may be down.

Obviously if the RequestDB is down, there is not a whole lot we can do but we can probably catch and handle the error better (try:/except: around the submit and a reported message perhaps?)

Target list doesn't always show the full list of newly ingested targets. These are being grouped by date. I suspect its sensitive to the seconds in the time. This list should just group by date and not time.

When running on the devel-eg branch with Debug=False using the runserver, it has trouble finding the static files needed and the CSS does not load:
tlister-linux{tlister} [311] [9:02] python manage.py runserver 172.16.4.3:8000 >&! Error_log

[15/Apr/2015 16:03:02] "GET / HTTP/1.1" 200 3069
Internal Server Error: /static/ingest/images/lcogt.png
Traceback (most recent call last):
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 148, in get_response
    response = callback(request, **param_dict)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/views/defaults.py", line 20, in page_not_found
    t = loader.get_template(template_name) # You need to create a 404.html template.
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 145, in get_template
    template, origin = find_template(template_name)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 138, in find_template
    raise TemplateDoesNotExist(name)
TemplateDoesNotExist: 404.html
Traceback (most recent call last):
  File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 151, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 226, in handle_uncaught_exception
    return callback(request, **param_dict)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/views/defaults.py", line 32, in server_error
    t = loader.get_template(template_name) # You need to create a 500.html template.
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 145, in get_template
    template, origin = find_template(template_name)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 138, in find_template
    raise TemplateDoesNotExist(name)
TemplateDoesNotExist: 500.html

Screenshot attached

I am using the stock settings.py and AFAIK have done the manage.py collectstatic - I am sure I am doing something stoopid but can you take a look ?

Tim

A new entry appeared today (2015-07-28) on the Goldstone schedule list:

2015 Oct 25-Nov 1 410777 2009 FD No Yes R PHA
which has a different number of columns than expected or seen before and results in a entry for (1) Ceres (!) rather than (410777) 2009 FD.
Will need to add extra tests and checks for this date range (which we don't currently make use of but likely will in the future to plan campaigns)

Currently we have a simple user and proposal model (by design) where all registered users can submit to any proposal. As the number of users and proposals grows, we are likely to need something more sophisticated.
Could you:

1. investigate how hard it would be integrate with the LCOGT proposal/user system and
2. see how much work it would be to have a user <-> proposal mapping system where specific users are associated with specific proposals.

To keep track of our requests a Block should be instantiated when a observations of an object are requested. At present this doesn't happen for an unknown reason, possibly that the code is silently failing.

The calculate ephemerides computation led to a "Bad Gateway" page, possibly due to timing out before finishing the computation. The schedule observations link had a difference of 15 arcmin in RA and 3.5 degrees in Dec between that reported here and that on the MPC page. The window of observation midpoint was also computed as a time when the object was only at +2 degrees altitude.

This currently follows the except part on too many occasions, erroneously resulting in a blank ranking.

In early January 2016, we ended up with a very large number (~800) of "new" targets. I suspect this is because the Previous NEOCP page does not have a year and we have to assume the current year. Edward, Can you investigate whether this is the case and what we can do about it?

Everything is lumped into the 'ingest' app. Investigate what is the simplest and clearest structure to allow this project to be easily maintainable.

The source measurements are not being added to if an object is updated on the NEOCP. This is due to a logic error in core.views.update_NEOCP_observations() which checks to see if a Body has been updated - unfortunately this will always be the case as the Body.update_time is set beforehand by update_NEOCP_orbit() before we get to update_NEOCP_observations()

When you try to submit observations to the scheduler I noticed we are getting:

error_type -> ProposalPermissionException
error_msg -> User tim.lister does not have access to proposal LCO2015A-009

I can't find where this error comes from so I suspect it is from reqdb.client.

We could save bandwidth, load on the MPC's servers and speed up ingest by 1) storing/using a last updated time on our Body's and 2) using the MPC NEOCP's 'Updated' column to check whether it has been more recently than when we last updated the corresponding Body

Objects which do not have a provisional name (primarily Goldstone objects) break with a 'Server Error' when trying to schedule them.

We need a form which will parse output from Astrometrica and save it as multiple Source Measurements. We also will need to display these Source Measurements in 80 char, monospaced format.

I noticed during the time I was at the IAU that the Previous NEO Confirmation Page Objects parsing had broken. Running the command manually produces:

Traceback (most recent call last):
  File "/var/www/apps/neoexchange/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/var/www/apps/neoexchange/core/management/commands/update_crossids.py", line 13, in handle
    objects = fetch_previous_NEOCP_desigs()
  File "/var/www/apps/neoexchange/astrometrics/sources_subs.py", line 131, in fetch_previous_NEOCP_desigs
    date = '('+provid_date[1] + ' ' + subitems[5] + ' ' + subitems[6]
IndexError: list index out of range

I suspect something is misformatted in the current page which is causing the code to break.

Add the VERSION setting into settings.py and display it on:

• dev server console
• home page

Eventually the site will be on the public web. We should require people to be logged in to do certain functions. We would like the portal to be as open as possible so to start with I suggest we only require log in for submitting observation requests.

We don't seem to have a good vision for what the homepage of NEO exchange is for. What are the actions for information we want displayed there? If this site is to become public then we should factor that in too.

Edward, Somehow my fix for badly behaving comets in astrometrics/ephem_subs.py in commit 1dae1e1 has been overwritten by your commit in d06c60b.
Can you investigate what happened and put the fix back?

The scheduling form needs the ability to alter the block length.

Observing windows that straddle the semester boundaries will be rejected as the proposal will not have time in the other semester. The windows should be truncated by the semester boundaries.

This is the details from the object causing us problems.

{'origin': u'M', 'slope': 0.15, 'epochofel': datetime.datetime(2015, 10, 25, 0, 0), 'provisional_packed': None, 'arc_length': 0.0, 'update_time': datetime.datetime(2015, 11, 1, 22, 8, 28), 'active': True, u'id': 3714L, 'discovery_date': datetime.datetime(2015, 11, 1, 19, 12), 'score': 100L, 'orbinc': 8.98042, 'num_obs': 5L, 'meananom': 345.87056, 'updated': False, 'abs_mag': 19.5, 'epochofperih': None, 'perihdist': None, 'provisional_name': u'EUHT950', 'source_type': u'U', 'longascnode': 7.8065, 'fast_moving': False, 'eccentricity': 0.5367056, 'argofperih': 47.03212, 'name': None, 'not_seen': 0.866, 'ingest': datetime.datetime(2015, 11, 1, 22, 20, 6), 'meandist': 1.9442854, 'elements_type': u'MPC_MINOR_PLANET', 'urgency': None}
(datetime.datetime(2015, 11, 2, 19, 46, 9, 314415), 5.190530360874316, -0.32658846765878347, 20.403257038447375, 1.5345465381470196, -1.8193723132912634, 71.286111111111111)

Our current functional tests for style and layout are brief. The tests should make sure we don't change some part of the CSS and make the site render poorly, or worse totally illegible.

Currently there are two failing tests in the NEOexchange test suite. One is the test I created in astrometrics.tests.test_time_subs.TestDegreesToHMS to demonstrate a bug in the rounding in the time/co-ordinate conversion functions.
The other is in core.tests.test_views.TestCheck_for_block:

======================================================================
FAIL: test_block_update_not_active (core.tests.test_views.TestCheck_for_block)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "/home/tlister/GIT/neoexchange/neoexchange/core/tests/test_views.py", line 753, in test_block_update_not_active
    self.assertFalse(resp)
AssertionError: True is not false

Not sure what the intent is here.

Currently when reporting a block via the button, it returns to the Home page and not the block page which I think would be better/more natural.

I noticed the cron job on the live site automatically updated on 12 June but hasn't updated since. The code seems to fail on core/management/commands/update_neocp_data.py on the live site but not using runserver.

It seemed like an obvious thing to do, so I added observation thumbnails on the Block detail page. At the moment they link to observations (and this is broken) but I can easily put a link to download the data from framedb or IPAC.

README.md needs more complete information.

• Environment vars usage
• Deployment
• Purpose
• Current version

Currently we have two ex-candidates, WK30CB1 and TFKT02, which have been given the same MPC designation of 2015 KW120. This object is also a Goldstone target but attempts to update that target list causes:

tlister-linux{tlister} [113] [16:42] python manage.py fetch_goldstone_targets
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/home/tlister/git/neoexchange_stable/neoexchange/ingest/management/commands/fetch_goldstone_targets.py", line 20, in handle
    update_MPC_orbit(obj_id, origin='G')
  File "/home/tlister/git/neoexchange_stable/neoexchange/ingest/views.py", line 460, in update_MPC_orbit
    body, created = Body.objects.get_or_create(name=obj_id)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/db/models/manager.py", line 134, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/db/models/query.py", line 445, in get_or_create
    return self.get(**lookup), False
  File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/db/models/query.py", line 368, in get
    % (self.model._meta.object_name, num, kwargs))
ingest.models.MultipleObjectsReturned: get() returned more than one Body -- it returned 2! Lookup parameters were {'name': '2015 KW120'}

It's not clear whether this is a correct result on the 'Previous NEOCP Page' (probably) or not, but in any case we need to handle this (In the old code this wasn't a problem as multiple Crossids could be made but in NEOx we need to handle it somehow by returning the last updated or designating one as the master etc )

When submitting requests to FTN/FTS the instrument config the instrument configs don't use the correct values.

When starting a new DB from scratch without any objects in the Body table , the home page view does not load producing a 500 error

There are currently two candidates on the NEOCP with provisional names P10rAyA and P10rAya, which have different enough predicted positions and orbits to be two different objects. I noticed today that only P10rAyA has been stored in NEOexchange, however it has the orbital elements, follow-up details, and predicted ephemeris for P10rAya (in other words it is P10rAya, but has been called P10rAyA within NEOexchange (as well as in observations taken at LSC on 2016 01 18 UT)). Since we have only captured one of these two objects with very similar names, we need to make sure the provisional names are case sensitive when stored in NEOexchange. Due to this problem, we have likely lost some NEO candidates. It seems these are likely to be all PanSTARRS candidates since the naming scheme they use the only one that would produce provisional names that are case sensitive. It's unclear how many candidates we've lost (probably not a huge number), but we may be able to get an estimate if someone digs through the PanSTARRs archive.

In order to support operations with NEO candidates where:

1. recoveries on the LCOGT Network need to be combined with the discovery and prior follow-up observations to confirm successful recovery and check astrometric residuals,
2. maintain access to the above data in the case of objects which are not NEOs which vanish from the NEOCP for at least a week
3. provide the foundation of the future infrastructure to store and report astrometric and photometric measurements made with the LCOGT NEO Follow-up Network

we need to able to support and perform the following tasks:

• create models to handle observed frames (happened in #46 ) and store source measurements
• design a data model that is able to handle the existing MPC 1992 80 column format (http://www.minorplanetcenter.net/iau/info/OpticalObs.html) and the future ADES XML standard (http://www.minorplanetcenter.net/mpec/K15/K15O06.html)
• create a parser and management command to ingest existing files of measurements for previously observed candidates
• synthesize and display MPC 80-column format lines from the stored source measurements (in #49 )
• create Frame entries from our observed FITS frames as part of the Block updating process
• Download and store observations from the NEOCP for candidates
• Handle S/s format satellite observations from e.g. C51 (NEOWISE)
• Handle and search for Bodies with a packed designations
• Allow upload of results from Astrometrica through either a web form or file reading (maybe) - covered in #49

Since the deployment of the extra_neocp_fields branch, extra fields are available to store more of the elements returned from fetch_mpcorbit (used for Goldstone and other targets). It would be useful to store more of these in the new Body fields.

Schedule windows currently only include time when it's dark, not the time when it's dark and the object is up.

If I run the code standalone with python manage.py runserver 172.16.4.3:8000 (or even without a specific IP), the main site is visible at http://172.16.4.3:8000 but the URLs to the targets, blocks and bodies have a 'neoexchange/' prefix (e.g. http://172.16.4.3:8000/neoexchange/target/) and so don't work.

For the 2015B semester, we have a small allocation of Target of Opportunity (ToO) time which will evict a running block from the telescopes and observe the new target within ~10 minutes. This should be supported in NEOexchange; things that are likely to be needed:

1. Add a 'Allowed to ToO' role to the user model
2. Add a 'Is this a ToO?' flag/tick box to submit forms
3. Add support for submitting ToOs to the RequestDB into the backend routines
4. Likely will need a field in the Block model and some sort of distinguishing icon/flag in the Block lists/results
5. It could be worth having an 'Observe this now!' button which works out where and when in the world it's best to do it/

The observation block for 2015 VO142 on 2015 11 20 UT at LSC was not marked as 'observed' within NEOexchange, even though we got 44 of the 44 images requested. This should be looked into.

As a note, this data was reduced and the astrometric measurements sent off to the MPC on 2015 11 20 21:29:00 UT, but since the block was not marked as 'observed' there is no ability to mark this as reported within NEOexchange.

This may be an effect caused by a very wide screen (my X1 laptop has 2560x1440 resolution apparently) but I am not seeing the Search box on the NEOexchange homepage and the edge of the pages are being cutoff (scrolling right doesn't reveal the rest either, it's just missing); see attached screenshot (hopefully)

The 'Recent Blocks Scheduled' part of the Body detail page is missing information about the actual blocks that have been scheduled e.g. https://lcogt.net/neoexchange/target/2556/ did not show the block I scheduled for CPT. This seems to be consistent across the half a dozen or so targets I scheduled today (2015-08-10)

Blocks need to be checked periodically to keep track of observation requests. This requires a call to the Request API with the block parameters to update the current status. I suggest management command and cron job for this.

The 'Recent blocks scheduled' section of the target/Body details page (e.g. https://lcogt.net/neoexchange/target/2973/) needs separate icons for observed and reported blocks. Currently it shows a watch/clock icon when it's scheduled and this changes to a 'thumb-up' when observed - there should be a separate true and false icon to report the 'Reported' status (maybe an icon of a document or a mail envelope?)