Comments (23)
wmwv
commented on September 27, 2024
Done for LSQ13cwp.
Next PTF 11mty.
from wiyn_lsst.
wmwv
commented on September 27, 2024
PTF 11mty processes. All 6 J, H images generate reasonable-seeming zeropoints.
from wiyn_lsst.
wmwv
commented on September 27, 2024
makeDiscreteSkyMap.py looks like what I want to use. It generates a map for a specific location (appropriate for a field-targeted survey), rather than a sky tiling (appropriate for a large-area survey).
But for each field, I need to append the creation of the field using --config doAppend=True
from wiyn_lsst.
wmwv
commented on September 27, 2024
Could also consider using makeDiscreteSkyMapRunner which is designed to accept an entire set of dataRefs at once.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Successfully generated H-band, J-band coadds for PTF11mty.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Had to specify which field to sort on in /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/skymap/14.0-4-g3609236+4/python/lsst/skymap/baseSkyMap.py, line 129:
distTractInfoList.sort(key=lambda x: x[0])
from wiyn_lsst.
wmwv
commented on September 27, 2024
Now failing trying to look up some information about the original Visit?
measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs
measureCoaddSources FATAL: Failed on dataId=DataId(initialdata={'tract': 1, 'patch': '0,0', 'field': 'PTF11mty', 'filter': 'J'}, tag=set()): OperationalError: no such table: raw_visit
Traceback (most recent call last):
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
result = task.run(dataRef, **kwargs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/multiBand.py", line 1098, in run
exposure.getWcs())
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/propagateVisitFlags.py", line 184, in run
ccdSources = butler.get("src", visit=int(v), ccd=int(c), immediate=True)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1408, in get
location = self._locate(datasetType, dataId, write=False)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1329, in _locate
location = repoData.repo.map(datasetType, dataId, write=write)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/repository.py", line 246, in map
loc = self._mapper.map(*args, **kwargs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/mapper.py", line 166, in map
return func(self.validate(dataId), write)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/cameraMapper.py", line 383, in mapClosure
return mapping.map(mapper, dataId, write)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 142, in map
actualId = self.need(iter(self.keyDict.keys()), dataId)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 278, in need
lookups = self.lookup(newProps, newId)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 242, in lookup
result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/registries.py", line 369, in lookup
cursor.execute(cmd, valueList)
sqlite3.OperationalError: no such table: raw_visit
The program exited via sys.exit(). Exit status: 2
> /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/bin/measureCoaddSources.py(2)<module>()
-> from lsst.pipe.tasks.multiBand import MeasureMergedCoaddSourcesTask
from wiyn_lsst.
wmwv
commented on September 27, 2024
The issue above is that PropagateVisitFlagsTask assumes that there is a ccd entry in the raw (or in this case stack) table. This is non-crazy, but not documented or otherwise enforced in the ingest step.
from wiyn_lsst.
wmwv
commented on September 27, 2024
-
I changed the
srcandtransformed_srcdefinitions so thattables: stackinstead oftables: raw. -
With some debugging print statements added, the relevant output is now:
REPODATA, DATASETTYPE, DATAID
RepoData(id=139934254811904,repoArgs=RepositoryArgs(root='/home/lsst/SweetSpot/DR1_data/tmp/test_dr1', cfgRoot=None, mapper=<class 'lsst.obs.wiyn.whircMapper.WhircMapper'>, mapperArgs=None, tags=set(), mode='rw', policy=None)cfg=RepositoryCfg(root='/home/lsst/SweetSpot/DR1_data/tmp/test_dr1', mapper=<class 'lsst.obs.wiyn.whircMapper.WhircMapper'>, mapperArgs={}, parents=[], policy=None),cfgOrigin=existing,cfgRoot=/home/lsst/SweetSpot/DR1_data/tmp/test_dr1,repo=<lsst.daf.persistence.repository.Repository object at 0x7f44fb8cc588>,parentRepoDatas=[],isV1Repository=True,role=output,parentRegistry=None) src DataId(initialdata={'visit': 0, 'ccd': -1}, tag=set())
CMD, VALUELIST
('SELECT DISTINCT field, seq, filter, night FROM stack_visit WHERE visit = ? AND ccd = ?', [0, -1])
measureCoaddSources FATAL: Failed on dataId=DataId(initialdata={'tract': 1, 'patch': '0,0', 'field': 'PTF11mty', 'filter': 'H'}, tag=set()): OperationalError: no such column: ccd
Traceback (most recent call last):
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
result = task.run(dataRef, **kwargs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/multiBand.py", line 1098, in run
exposure.getWcs())
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/propagateVisitFlags.py", line 182, in run
for ccdRecord in ccdInputs:
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1410, in get
location = self._locate(datasetType, dataId, write=False)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1331, in _locate
location = repoData.repo.map(datasetType, dataId, write=write)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/repository.py", line 246, in map
loc = self._mapper.map(*args, **kwargs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/mapper.py", line 166, in map
return func(self.validate(dataId), write)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/cameraMapper.py", line 383, in mapClosure
return mapping.map(mapper, dataId, write)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 142, in map
actualId = self.need(iter(self.keyDict.keys()), dataId)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 278, in need
lookups = self.lookup(newProps, newId)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 242, in lookup
result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/registries.py", line 371, in lookup
cursor.execute(cmd, valueList)
sqlite3.OperationalError: no such column: ccd
from wiyn_lsst.
wmwv
commented on September 27, 2024
I will try skipping PropagateVisitFlags for now with --doPropagateFlags=False
from wiyn_lsst.
wmwv
commented on September 27, 2024
Following instructions at https://pipelines.lsst.io/getting-started/photometry.html
For PTF11mty coadds have now successfully:
- Detected
- Merged Detections
- Measured
- Merged Measurements
- Forced photometry at merged object locations.
from wiyn_lsst.
wmwv
commented on September 27, 2024
The Forced photometry spews errors for all of the tracts that don't exist, but still appears to generate output for the one tract+patch that does exist.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Coadds look good. Example for LSQ13cwp
from wiyn_lsst.
wmwv
commented on September 27, 2024
mag err vs. mag looks reasonable.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Incorporated the deepCoadd_ref table, including star/galaxy separation information, but it's not very good. Obvious galaxies are marked as stars and vice versa.
from wiyn_lsst.
wmwv
commented on September 27, 2024
When running all of the fields, I was getting failures where processCcd.py was exiting. Because it was running 8 jobs in parallel it was a bit tough to diagnose exactly what was happening, but I think it's the K-band images that were failing. These are registered as filter KS. I'm re-running the full list without the K-band
from wiyn_lsst.
wmwv
commented on September 27, 2024
It looks like jobs are hanging on something. I see 9 python processes for each invocation of run_processCcd.sh. I think these aren't getting output to the log. Will debug in -j 1 tomorrow.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Tried a sample image that previously failed: P161-D_A_KS_20121122
WIYN=${DR1BASE}/tmp
REPO=${WIYN}/test_dr1
processCcd.py "${REPO}" --rerun processCcdKS \
--id field=P161-D seq=A filter=KS night=20121122 expnum=528 \
--configfile config/processCcd.py
This failed as it had yesterday with:
processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 173/943 sources
processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 30/30 references
processCcd.calibrate.photoCal.match INFO: Matched 0 from 173/943 input and 30/30 reference sources
processCcd.calibrate.photoCal.reserve INFO: Reserved 0/0 sources
processCcd FATAL: Failed on dataId=DataId(initialdata={'field': 'P161-D', 'seq': 'A', 'filter': 'KS', 'night': 20121122, 'expnum': '528'}, tag=set()): RuntimeError: No matches to use for photocal
Traceback (most recent call last):
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
result = task.run(dataRef, **kwargs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/processCcd.py", line 199, in run
icSourceCat=charRes.sourceCat,
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/calibrate.py", line 433, in run
icSourceCat=icSourceCat,
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/calibrate.py", line 535, in calibrate
photoRes = self.photoCal.run(exposure, sourceCat=sourceCat, expId=exposureIdInfo.expId)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/photoCal.py", line 464, in run
raise RuntimeError("No matches to use for photocal")
RuntimeError: No matches to use for photocal
So this means the photometry calibration code is not finding the matches between the sources and the reference catalog.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Looking more closely at the stack I saw that the WCS solution was slightly off by ~0.5 arcsec.
I ran it with astrometric solving and it succeeded:
processCcd.py "${REPO}" --rerun processCcdKS \
--id field=P161-D seq=A filter=KS night=20121122 expnum=528 \
--configfile config/processCcd.py \
--config calibrate.doAstrometry=True
processCcd.calibrate.astromRefObjLoader INFO: Loading reference objects using center IcrsCoord(105.2138898, 48.4906848) and radius 0.06316164018914798 deg
processCcd.calibrate.astromRefObjLoader INFO: Loaded 33 reference objects
processCcd.calibrate.astrometry.matcher INFO: Purged 931 sources, leaving 12 good sources
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher INFO: Matched 12 sources
processCcd.calibrate.astrometry.matcher INFO: Purged 931 sources, leaving 12 good sources
processCcd.calibrate.astrometry.matcher INFO: Matched 12 sources
processCcd.calibrate.astrometry.matcher INFO: Purged 931 sources, leaving 12 good sources
processCcd.calibrate.astrometry.matcher INFO: Matched 12 sources
processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 12 matches with scatter = 0.004 +- 0.003 arcsec
processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 173/943 sources
processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 34/34 references
processCcd.calibrate.photoCal.match INFO: Matched 12 from 173/943 input and 34/34 reference sources
processCcd.calibrate.photoCal.reserve INFO: Reserved 0/12 sources
processCcd.calibrate.photoCal INFO: Not applying color terms because config.applyColorTerms is None and data is not available and photoRefCat is provided
processCcd.calibrate.photoCal INFO: Magnitude zero point: 26.380040 +/- 0.000343 from 12 stars
processCcd.calibrate INFO: Photometric zero-point: 26.380040
from wiyn_lsst.
wmwv
commented on September 27, 2024
I had previously disabled the astrometric fitting because it was failing to find matches for images that should have been trivially simple with WCS solutions that were already good to ~0.1 arcsec.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Hmmm... indirectly related to the low success rate of processing only ~150 out of ~700 stacks last night (with no astrom fitting), I realize that I'm running this in a Docker VM with only 2GB. Running 8 simultaneous processing is probably too much.
from wiyn_lsst.
wmwv
commented on September 27, 2024
Fitting astrometry failed for P525-E_B_H_20131019
from wiyn_lsst.
wmwv
commented on September 27, 2024
I'm going to merge the present state to master. Then block on further progress until all stacks can be processed with processCcd.py #16 .
from wiyn_lsst.
Related Issues (20)
- Adapt to fully use `obs_wiyn` HOT 1
- Create HTM catalogs for SweetSpot calibration HOT 1
- Solve "Too Many CR pixels" errors; presumably by setting smaller initial PSF guess HOT 1
- Do forced-photometry on stacks based on coadd detections for all objects HOT 1
- Run subtractions for DR1 fields with references. HOT 1
- Run processCcd for all DR1 stacks HOT 5
- Run coadds for all stacks HOT 20
- Run subtraction for one field. HOT 1
- Extract lightcurve for one field: LSQ13cwp HOT 4
- Extract lightcurve from subtractions for isolated SN
- Create individual-image lightcurves for 33 DR1 supernova HOT 12
- Switch to including `expnum` in filenames HOT 2
- Figure out why I need to specify `expnum` in subtractions.
- Make sn_lightcurve.py use same Butler object HOT 1
- Identify why some fields have multiple patches in the tract.
- Investigate LSQ13cwp, LSQ13cmt, iPTF13dkl calexp forced-photometry lightcurves HOT 2
- Run calexp forced photometry for non-host-contaminated SNe HOT 2
- Calibrate all DR2 images. HOT 3
- The filename mapper is not figuring out what to call the file in subtraction routine HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wiyn_lsst.