lcogt / neoexchange Goto Github PK
View Code? Open in Web Editor NEWNEO observing portal
License: GNU General Public License v3.0
NEO observing portal
License: GNU General Public License v3.0
When testing NEOx we've accidentally sent things to the scheduler. When using the dev server (or even running tests) we should automatically disable submitting to the scheduler.
My code which is intended to read a NEOCP 1-line orbit file from disk and use the Body
from the models.py
and then tries to call save_and_make_revision()
from views.py
is dying horribly with the following traceback:
Since the implementation of the site URL prefix, the admin site does not redirect correctly after login. After a successful login, it redirects to:
http://docknode06:8200/admin/login/?next=/admin/
producing a 404 Not Found nginx/1.6.3 error.
It would be helpful, in the overall Targets view (/target/) and Target-specific view (/target/XXXX/) if the prior provisional designation was available for objects which have a new final designation. In the old NEO site, I formatted the cross idents in a single column as 'P10luqgh-->2015 LB2' but this is only a suggestion.
Blocks page is long and in need of either pagination or something more complicated (e.g. pulling in all active blocks which wouldn't be in the first page).
We wish to include those targets that are on the Arecibo target list (http://www.naic.edu/~pradar/) as these 1) may not be on the Goldstone list and 2) DSS-14 will be down for maintenance for 2016 Feb-May.
Unresolved question: do we want a specific origin type for objects that are on both lists (currently common objects will ping-pong from origin=Arecibo to origin=Goldstone and back again depending on which was the last management command).
Today (2015-0-29 21:25 UTC), I tried to submit a block for 2015 SZ2 to CPT during the time the RequestDB had been taken down for upgrades which produced a ServerError (and logged me out so I couldn't see the messages). From /var/www/apps/neoexchange/neox.log
:
[29/Sep/2015 21:23:48] ERROR Internal Server Error: /neoexchange/schedule/3161/confirm/
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
return handler(request, *args, **kwargs)
File "./core/views.py", line 212, in post
return super(ScheduleSubmit, self).post(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 215, in post
return self.form_valid(form)
File "./core/views.py", line 216, in form_valid
tracking_num, sched_params = schedule_submit(form.cleaned_data, target)
File "./core/views.py", line 308, in schedule_submit
tracking_number, resp_params = submit_block_to_scheduler(body_elements, params)
File "./astrometrics/sources_subs.py", line 662, in submit_block_to_scheduler
response_data = client.submit(user_request)
File "/usr/lib/python2.7/site-packages/reqdb/client.py", line 261, in submit
response_data = self._submit_to_endpoint(post_req, debug)
File "/usr/lib/python2.7/site-packages/reqdb/client.py", line 409, in _submit_to_endpoint
raise ConnectionError(msg)
ConnectionError: Received an error from server: [Errno 111] Connection refused
Your url may be wrong, or the endpoint at 'http://scheduler1.lco.gtn/requestdb/' may be down.
Obviously if the RequestDB is down, there is not a whole lot we can do but we can probably catch and handle the error better (try:/except:
around the submit and a reported message perhaps?)
Target list doesn't always show the full list of newly ingested targets. These are being grouped by date. I suspect its sensitive to the seconds in the time. This list should just group by date and not time.
When running on the devel-eg branch with Debug=False
using the runserver, it has trouble finding the static files needed and the CSS does not load:
tlister-linux{tlister} [311] [9:02] python manage.py runserver 172.16.4.3:8000 >&! Error_log
[15/Apr/2015 16:03:02] "GET / HTTP/1.1" 200 3069
Internal Server Error: /static/ingest/images/lcogt.png
Traceback (most recent call last):
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 148, in get_response
response = callback(request, **param_dict)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/views/defaults.py", line 20, in page_not_found
t = loader.get_template(template_name) # You need to create a 404.html template.
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 145, in get_template
template, origin = find_template(template_name)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 138, in find_template
raise TemplateDoesNotExist(name)
TemplateDoesNotExist: 404.html
Traceback (most recent call last):
File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
response = self.get_response(request)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 151, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 226, in handle_uncaught_exception
return callback(request, **param_dict)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/views/defaults.py", line 32, in server_error
t = loader.get_template(template_name) # You need to create a 500.html template.
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 145, in get_template
template, origin = find_template(template_name)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/template/loader.py", line 138, in find_template
raise TemplateDoesNotExist(name)
TemplateDoesNotExist: 500.html
I am using the stock settings.py
and AFAIK have done the manage.py collectstatic
- I am sure I am doing something stoopid but can you take a look ?
Tim
A new entry appeared today (2015-07-28) on the Goldstone schedule list:
2015 Oct 25-Nov 1 410777 2009 FD No Yes R PHA
which has a different number of columns than expected or seen before and results in a entry for (1) Ceres (!) rather than (410777) 2009 FD.
Will need to add extra tests and checks for this date range (which we don't currently make use of but likely will in the future to plan campaigns)
Currently we have a simple user and proposal model (by design) where all registered users can submit to any proposal. As the number of users and proposals grows, we are likely to need something more sophisticated.
Could you:
To keep track of our requests a Block should be instantiated when a observations of an object are requested. At present this doesn't happen for an unknown reason, possibly that the code is silently failing.
The calculate ephemerides computation led to a "Bad Gateway" page, possibly due to timing out before finishing the computation. The schedule observations link had a difference of 15 arcmin in RA and 3.5 degrees in Dec between that reported here and that on the MPC page. The window of observation midpoint was also computed as a time when the object was only at +2 degrees altitude.
This currently follows the except part on too many occasions, erroneously resulting in a blank ranking.
In early January 2016, we ended up with a very large number (~800) of "new" targets. I suspect this is because the Previous NEOCP page does not have a year and we have to assume the current year. Edward, Can you investigate whether this is the case and what we can do about it?
Everything is lumped into the 'ingest' app. Investigate what is the simplest and clearest structure to allow this project to be easily maintainable.
The source measurements are not being added to if an object is updated on the NEOCP. This is due to a logic error in core.views.update_NEOCP_observations()
which checks to see if a Body has been updated - unfortunately this will always be the case as the Body.update_time
is set beforehand by update_NEOCP_orbit()
before we get to update_NEOCP_observations()
When you try to submit observations to the scheduler I noticed we are getting:
error_type -> ProposalPermissionException
error_msg -> User tim.lister does not have access to proposal LCO2015A-009
I can't find where this error comes from so I suspect it is from reqdb.client
.
We could save bandwidth, load on the MPC's servers and speed up ingest by 1) storing/using a last updated time on our Body's and 2) using the MPC NEOCP's 'Updated' column to check whether it has been more recently than when we last updated the corresponding Body
Objects which do not have a provisional name (primarily Goldstone objects) break with a 'Server Error' when trying to schedule them.
We need a form which will parse output from Astrometrica and save it as multiple Source Measurements. We also will need to display these Source Measurements in 80 char, monospaced format.
I noticed during the time I was at the IAU that the Previous NEO Confirmation Page Objects parsing had broken. Running the command manually produces:
Traceback (most recent call last):
File "/var/www/apps/neoexchange/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
output = self.handle(*args, **options)
File "/var/www/apps/neoexchange/core/management/commands/update_crossids.py", line 13, in handle
objects = fetch_previous_NEOCP_desigs()
File "/var/www/apps/neoexchange/astrometrics/sources_subs.py", line 131, in fetch_previous_NEOCP_desigs
date = '('+provid_date[1] + ' ' + subitems[5] + ' ' + subitems[6]
IndexError: list index out of range
I suspect something is misformatted in the current page which is causing the code to break.
Add the VERSION
setting into settings.py
and display it on:
Eventually the site will be on the public web. We should require people to be logged in to do certain functions. We would like the portal to be as open as possible so to start with I suggest we only require log in for submitting observation requests.
We don't seem to have a good vision for what the homepage of NEO exchange is for. What are the actions for information we want displayed there? If this site is to become public then we should factor that in too.
The scheduling form needs the ability to alter the block length.
Observing windows that straddle the semester boundaries will be rejected as the proposal will not have time in the other semester. The windows should be truncated by the semester boundaries.
This is the details from the object causing us problems.
{'origin': u'M', 'slope': 0.15, 'epochofel': datetime.datetime(2015, 10, 25, 0, 0), 'provisional_packed': None, 'arc_length': 0.0, 'update_time': datetime.datetime(2015, 11, 1, 22, 8, 28), 'active': True, u'id': 3714L, 'discovery_date': datetime.datetime(2015, 11, 1, 19, 12), 'score': 100L, 'orbinc': 8.98042, 'num_obs': 5L, 'meananom': 345.87056, 'updated': False, 'abs_mag': 19.5, 'epochofperih': None, 'perihdist': None, 'provisional_name': u'EUHT950', 'source_type': u'U', 'longascnode': 7.8065, 'fast_moving': False, 'eccentricity': 0.5367056, 'argofperih': 47.03212, 'name': None, 'not_seen': 0.866, 'ingest': datetime.datetime(2015, 11, 1, 22, 20, 6), 'meandist': 1.9442854, 'elements_type': u'MPC_MINOR_PLANET', 'urgency': None}
(datetime.datetime(2015, 11, 2, 19, 46, 9, 314415), 5.190530360874316, -0.32658846765878347, 20.403257038447375, 1.5345465381470196, -1.8193723132912634, 71.286111111111111)
Our current functional tests for style and layout are brief. The tests should make sure we don't change some part of the CSS and make the site render poorly, or worse totally illegible.
Currently there are two failing tests in the NEOexchange test suite. One is the test I created in astrometrics.tests.test_time_subs.TestDegreesToHMS
to demonstrate a bug in the rounding in the time/co-ordinate conversion functions.
The other is in core.tests.test_views.TestCheck_for_block
:
======================================================================
FAIL: test_block_update_not_active (core.tests.test_views.TestCheck_for_block)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "/home/tlister/GIT/neoexchange/neoexchange/core/tests/test_views.py", line 753, in test_block_update_not_active
self.assertFalse(resp)
AssertionError: True is not false
Not sure what the intent is here.
Currently when reporting a block via the button, it returns to the Home page and not the block page which I think would be better/more natural.
I noticed the cron job on the live site automatically updated on 12 June but hasn't updated since. The code seems to fail on core/management/commands/update_neocp_data.py
on the live site but not using runserver
.
It seemed like an obvious thing to do, so I added observation thumbnails on the Block detail page. At the moment they link to observations (and this is broken) but I can easily put a link to download the data from framedb or IPAC.
README.md needs more complete information.
Currently we have two ex-candidates, WK30CB1 and TFKT02, which have been given the same MPC designation of 2015 KW120. This object is also a Goldstone target but attempts to update that target list causes:
tlister-linux{tlister} [113] [16:42] python manage.py fetch_goldstone_targets
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/home/tlister/git/neoexchange_stable/neoexchange/ingest/management/commands/fetch_goldstone_targets.py", line 20, in handle
update_MPC_orbit(obj_id, origin='G')
File "/home/tlister/git/neoexchange_stable/neoexchange/ingest/views.py", line 460, in update_MPC_orbit
body, created = Body.objects.get_or_create(name=obj_id)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/db/models/manager.py", line 134, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/db/models/query.py", line 445, in get_or_create
return self.get(**lookup), False
File "/home/tlister/python/neoexchange_venv/lib/python2.7/site-packages/django/db/models/query.py", line 368, in get
% (self.model._meta.object_name, num, kwargs))
ingest.models.MultipleObjectsReturned: get() returned more than one Body -- it returned 2! Lookup parameters were {'name': '2015 KW120'}
It's not clear whether this is a correct result on the 'Previous NEOCP Page' (probably) or not, but in any case we need to handle this (In the old code this wasn't a problem as multiple Crossids could be made but in NEOx we need to handle it somehow by returning the last updated or designating one as the master etc )
When submitting requests to FTN/FTS the instrument config the instrument configs don't use the correct values.
When starting a new DB from scratch without any objects in the Body table , the home page view does not load producing a 500 error
There are currently two candidates on the NEOCP with provisional names P10rAyA and P10rAya, which have different enough predicted positions and orbits to be two different objects. I noticed today that only P10rAyA has been stored in NEOexchange, however it has the orbital elements, follow-up details, and predicted ephemeris for P10rAya (in other words it is P10rAya, but has been called P10rAyA within NEOexchange (as well as in observations taken at LSC on 2016 01 18 UT)). Since we have only captured one of these two objects with very similar names, we need to make sure the provisional names are case sensitive when stored in NEOexchange. Due to this problem, we have likely lost some NEO candidates. It seems these are likely to be all PanSTARRS candidates since the naming scheme they use the only one that would produce provisional names that are case sensitive. It's unclear how many candidates we've lost (probably not a huge number), but we may be able to get an estimate if someone digs through the PanSTARRs archive.
In order to support operations with NEO candidates where:
we need to able to support and perform the following tasks:
S/s
format satellite observations from e.g. C51
(NEOWISE)Since the deployment of the extra_neocp_fields
branch, extra fields are available to store more of the elements returned from fetch_mpcorbit
(used for Goldstone and other targets). It would be useful to store more of these in the new Body fields.
Schedule windows currently only include time when it's dark, not the time when it's dark and the object is up.
If I run the code standalone with python manage.py runserver 172.16.4.3:8000
(or even without a specific IP), the main site is visible at http://172.16.4.3:8000 but the URLs to the targets, blocks and bodies have a 'neoexchange/' prefix (e.g. http://172.16.4.3:8000/neoexchange/target/) and so don't work.
For the 2015B semester, we have a small allocation of Target of Opportunity (ToO) time which will evict a running block from the telescopes and observe the new target within ~10 minutes. This should be supported in NEOexchange; things that are likely to be needed:
The observation block for 2015 VO142 on 2015 11 20 UT at LSC was not marked as 'observed' within NEOexchange, even though we got 44 of the 44 images requested. This should be looked into.
As a note, this data was reduced and the astrometric measurements sent off to the MPC on 2015 11 20 21:29:00 UT, but since the block was not marked as 'observed' there is no ability to mark this as reported within NEOexchange.
This may be an effect caused by a very wide screen (my X1 laptop has 2560x1440 resolution apparently) but I am not seeing the Search box on the NEOexchange homepage and the edge of the pages are being cutoff (scrolling right doesn't reveal the rest either, it's just missing); see attached screenshot (hopefully)
The 'Recent Blocks Scheduled' part of the Body detail page is missing information about the actual blocks that have been scheduled e.g. https://lcogt.net/neoexchange/target/2556/ did not show the block I scheduled for CPT. This seems to be consistent across the half a dozen or so targets I scheduled today (2015-08-10)
Blocks need to be checked periodically to keep track of observation requests. This requires a call to the Request API with the block parameters to update the current status. I suggest management command and cron job for this.
The 'Recent blocks scheduled' section of the target/Body details page (e.g. https://lcogt.net/neoexchange/target/2973/) needs separate icons for observed and reported blocks. Currently it shows a watch/clock icon when it's scheduled and this changes to a 'thumb-up' when observed - there should be a separate true and false icon to report the 'Reported' status (maybe an icon of a document or a mail envelope?)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.