This tool subsamples Postgres database of ENCODE/IGVF servers based on a subsampling rule JSON file.
This file defines subsampling rule(s) for each profile (e.g. experiment
for ENCODE, measurement_set
for IGVF). Multiple rules are allowed for each profile. Here is an example for ENCODE.
{
"file": [
{
"subsampling_min": 100,
"subsampling_rate": 1e-03
}
],
"experiment": [
{
"subsampling_min": 3,
"subsampling_rate": 1e-05,
"subsampling_cond": {
"assay_term_name": "ATAC-seq"
}
},
{
"subsampling_min": 5,
"subsampling_rate": 1e-05,
"subsampling_cond": {
"assay_term_name": "ChIP-seq"
}
}
]
}
A rule is consist of subsampling_min
, subsampling_rate
and subsampling_cond
(optional). See the following example of experiment
profile of ENCODE.
{
"subsampling_min": 5,
"subsampling_rate": 1e-05,
"subsampling_cond": {
"assay_term_name": "ChIP-seq"
}
}
subsampling_min
defines the minimum number of objects in the profile after subsampling. It's bound to the actual number of objects. i.e. takingMIN(number_of_objects, subsampling_min)
.subsampling_rate
defines the minimum number of objects as total (respectingsubsampling_cond
if defined) number of objects in the profile multiplied by the rate. MAX of these two values will be taken as the final number of subsampled objects in the profile.subsampling_cond
is a JSON object that defines conditions for the rule. For the above example, this will only subsample objects with a propertyassay_term_name
defined asChIP-seq
. You can use any valid property in a profile. See profile's schema JSON to find such property.
There are currently 12548 ChIP-seq
experiments and it will subsample 12548 objects down to MAX(5, 1e-05*12548) = 5
.
For the case of file
profile in the above example, there are currently 1458539 file
objects on ENCODE. So it will subsample 1458539
objects down to MAX(100, 1e-03 * 1458539) = 1458
.
You can have multiple rules under a single profile. See the case of experiment
profile in the above example. It will include at least 3 ATAC-Seq
experiments and 5 ChIP-seq
experiments.
IMPORTANT: Some users and their access_keys are important to run a server. Therefore, two examples keep ALL
user
andaccess_key
for subsampling. i.e."subsampling_min": 1000000
is defined foruser
andaccess_key
.
Examples keep ALL users after subsampling.
Install postgresql dev library on your system.
# apt
$ sudo apt-get install libpq-dev
# yum
$ sudo yum install postgresql-devel
See this document for details.
See this document for details.