dmwm / dbs Goto Github PK

Original TRAC ticket 927 reported by yuyi
Script for migrate DBS2 datasets into DBS3. Need to define what can be directly migrate and what need to be modified. Some of the dataset , such as old PromptReco data in DBS2 cannot be automatically migrate into DBS3 due to the mismatching global tag, dataset and Algorithm config.

Original TRAC ticket 699 reported by mnorman
Listing datasets in DBS3 through the API appears to be unreliable when presented with the dataset name as a full path.

Original TRAC ticket 360 reported by afaq
We need to review exception messages, either the wmcore will provide a mechanism to logging error messages with an identifier (GUID etc.) or DBS will do such a thing. In case of errors, clients will be send a message with this identifier

Original TRAC ticket 698 reported by mnorman
I can upload all the blocks, but only 60-90% of the files. This is a very odd error that needs looking into (and may very well be on my end of the BlockInsert code).

Original TRAC ticket 1073 reported by metson
Each REST method should have a clearly documented set of roles/groups that are allowed to call them, to make adding in the security simpler.

Original TRAC ticket 1074 reported by sryu
currently dbs/fileparents call only takes file lfn as an argument. This will be too many network call when workqueue needs the parentage information for all the files in a given block.
Workqueue only needs the list of parent file lfns by each file not other details as returned results. [{ 'file_lfn_1': [parent_file_1, parent_file2], ...}]

Original TRAC ticket 1057 reported by yuyi
We need to summary over number of events, number of files, number of lumi sections and number of blocks for a given dataset or a block. Note that both dataset and block are fully named.

Original TRAC ticket 362 reported by afaq
"some text bit"+acquisitionEra+ProcessingVersion will make the name of processed dataset.

Original TRAC ticket 435 reported by afaq
As per discussion during DBS review, add 'scenario' to output module configuration table.

Original TRAC ticket 1271 reported by yuyi
We need api to list all the possible dataset access types on DBS.

Original TRAC ticket 358 reported by afaq
at the moment, it expects to insert files in same block.

Original TRAC ticket 1135 reported by giffels
Patch to replace deprecated addMethod by private _addMethod in DBS Code.

Original TRAC ticket 964 reported by yuyi
update the DBSBlock/updateStatus API to set the last_modification_date to sysdate when set open_for_writing=0.

Original TRAC ticket 694 reported by mnorman
For extensive review before commit

Original TRAC ticket 696 reported by mnorman
Check to make sure we can't break SQL using malformed keys in the generic InsertTable implementation of DBS3.

Original TRAC ticket 436 reported by valya
Hi, in DAS I need to invoke URL with a set of parameters which is unknown a-priory and defined by users. I define a map and set allowed parameters to handle this case. If parameter is not provided by users it is assigned as '_'. I'm not sure if current implementation can handle this case for all APIs. I tried datasets and get three results based on provided set of parameters:

1. if I don't pass any parameter I get full list of dataset [ok]
2. if I pass dataset=_ I get full list of dataset [ok]
3. if I pass dataset=&release_name= I get empty list [not ok]
4. if I pass dataset=&run_num=&release_name=* I get 500 internal server error [not ok]

• turns out that run_num needs to be int, not *

Since some APIs can have large list of parameters we have two choices:

1. allow passing * for parameter which are not in use and it can be ignored by DBS
2. use full set of permutations of parameters to get possible set of allowed URLs to be used, e.g.
   http://.../datasets
   http://.../datasets?dataset=
   http://.../datasets?dataset=&release_name=
   etc.

Before making any request I want to know your opinion on this subject, obviously my preference is #1, but I can write maps for any permutation of parameters too, even though it will be very long list.

Thanks,
Valentin.

Original TRAC ticket 1222 reported by yuyi
We need to improve our APIs to allow the user to specify arguments using lower or mixed cases, such as dataset access type and file types.

Original TRAC ticket 437 reported by afaq
We finally decided not to have 'site' information in dBS, so we just drop this.

Original TRAC ticket 929 reported by yuyi
1.Add global tag into unique constraint of output_module_configs table based on the discussion with Dirk to store PromtReco data.
2. PRCOESSED_DS_NAME to be changed from varchar2(100) to varchart2(200) because some of the name in DBS2 it very very long.

Original TRAC ticket 359 reported by afaq
instead of insertFiles(), wmagent etc would insert everything-block into DBS, we need to "expose" that functionality through writer-model.

--putBlock() API will do the job.

Original TRAC ticket 1203 reported by yuyi
We need carefully update DBSBlockInsert API to reflect the schema change by adding global_tag as part of unique key. In addition, we need add checks on file output module config has to be part of its dataset's config.

Original TRAC ticket 417 reported by metson
Since DBS3 removes certain fields/tables it is necessary to document where the corresponding information will be accessible. Once completed a DAS ticket should be opened to track the availability of a DAS-interface to that information.

Original TRAC ticket 1005 reported by yuyi
Need to add authentication to securing DBS.

Original TRAC ticket 1004 reported by yuyi
As discussed with Valentin regarding Oli's requirement on search blocks by data tier, it is a better way to expand the listBlock api to allow min and max createion_date search in order to reduce the amount of data oracle need to scan.

Original TRAC ticket 357 reported by metson
Include state transition diagram

Original TRAC ticket 965 reported by yuyi
We assume that all files will have the same create_date as their block. So the API will not set it for individual files. The same thing goes with create_by column in files table.

Original TRAC ticket 1277 reported by yuyi
Requested by Valentin, we need to rewrite the result to have key-value pairs and include user input in the return.

Original TRAC ticket 930 reported by yuyi
Update everything due to the schema change in output module configs table.

Original TRAC ticket 760 reported by yuyi
DBS3 tests are out of date due to code and schema changes after the review, In addition, new business object DBSBlockInsert needs to be added to the unittests too.

Original TRAC ticket 363 reported by afaq
...and i forgot what table we decided it to go into ?

We will need to tag datasets from PREP requests with the PREP ID so that they can cross link the datasets to/from the request.

Not sure if this was done already or still open, so updating & refreshing this ticket.

Original TRAC ticket 1130 reported by liangd
In ticket 1004, only listBlock by date was disccussed,
Will ListDataset by date be support as well?

Original TRAC ticket 556 reported by yuyi
update schema following the DBS3 code review recommandation and code changes dut to schema uodate

Original TRAC ticket 928 reported by yuyi
Need to compare DBS3 against the source DBS2 after the migration.

Original TRAC ticket 1055 reported by yuyi
Need to write/update DBS3 API doc. Document the required user inputs for each API.

Original TRAC ticket 761 reported by yuyi
DAO has recently changed. Require to update the unittests.

Original TRAC ticket 1139 reported by mnorman
In migration, we've received an error of the following type:

DbsExecutionError:DBS Server Raised An Error:
API Invoked insertDatasetContents
Servlet Error,Already closed.

By invoking dbsMigrateBlock(). We were under the impression that migration of duplicate blocks should be ignored quietly, so as to prevent the uploader from halting due to accidental re-migration. Is this not correct?

If this is not correct, could the code be altered so that dbsMigrateBlock raises a distinct exception for duplication errors? Something we could catch and handle separately?

Original TRAC ticket 948 reported by valya
As been discussed in #938, the users need to be able to look-up dataset based on their status. Moreover the status should always be defaulted to VALID. Please review discussion and decide if status can be added into DBS3 listDatasets api.

Original TRAC ticket 361 reported by afaq
Global tag can change during production of a dataset, so it should move to "configuration" from dataset.

Original TRAC ticket 1144 reported by valya
I came across a concrete API example where the output for input query is ambiguous. For example,

https://cms-xen40.fnal.gov/DBS/runs?maxrun=1234&minrun=0&dataset=/*/unittest_web_dataset_2/GEN-SIM-RAW

yields the following results

[{u'run_num': 1}, {u'run_num': 6}, ...]

which does not allow to understand which run correspond to which dataset. So this should be fixed in a form

[{u'run_num': 1, 'dataset': '/a/b/c'}, {u'run_num': 6, 'dataset':'/c/d/e'}, ... ]

Please check across '''ALL''' APIs that you provide non-ambiguous output for input query. Basically the APIs which accept patterns must check the pattern and if it exists it must append it to the output rows.

Original TRAC ticket 456 reported by yuyi
my first svn example

Original TRAC ticket 1138 reported by liangd
failed to find the way to support this query in DBSReaderModel.
From the dbs3 schema, it seems the solution is :
provide a list on TABLE FILE_LUMIS.

Currently the only way to do this is:
RUNS => DATASET => FLIES => LUMIS

We need a RUNLUMI dao, listrunlumi bussiness, upto a API in DBSReaderModel.

Original TRAC ticket 1008 reported by liangd
The reason for this question is that if DBS3 don't support this kind of query any more.
Then DAS will not make efforts to support it, even currenty the user could do such query in DBS2.

I didn't find a api we can specify created_time params.
Raise this question to make sure.

Original TRAC ticket 697 reported by mnorman
Check DBS3 insertion code for more silent failures - failures where the insertion fails to write in but does not raise an exception passing back. This is probably going to be a fairly long and drawn out affair, and a lot of it will be done by finding bad things in running code.

Also, we'll need to arrive at a convention as to how errors are communicated through the API.

Original TRAC ticket 850 reported by yuyi
DBS3 server started by wmcoreD is dead after a few days of running w/o any notice. No error or message can be found in the server log file. The server life span is various from a few days to a week so. We have seen this for long time. Now it is time to investigate and fix the problem.

Original TRAC ticket 1163 reported by yuyi
new/changed APIS need new unit tests at least at web layer. refer to ticket 948, 1008, 1057, 1074, 1004 for details on what needs to be tested.

Original TRAC ticket 695 reported by mnorman
Create a permanent DBS3 Test Server that we can run tests against.

Original TRAC ticket 649 reported by metson
DBS should print standard logging messages.

Original TRAC ticket 438 reported by afaq
Simple removal from schema and double check for any code references.

Original TRAC ticket 1276 reported by yuyi
Valentin requested a api that can list all the version or a particular one, such as url/releases?release_version=CMSSW_2_0_8.

Original TRAC ticket 418 reported by metson
The cmsweb cluster requires deploy and management scripts. These need to be written for DBS3.