rglab / cytoml Goto Github PK

Not sure if this is new in vX, flowJo simply creates Not gates first and use And gate to create all boolean combinations.

It is not compatible with openCyto where the generic booleanFilter handles all 3 logical operations &, | , ! in a single boolean expression.

In order to generate a flowJo workspace, we have to either convert booleanFilter to these basic gate types when the GatingSet2flowJo is called or do it when booleanFilter was added to GatingSet in the first place.

We've been mostly dealing with the analysis using asinhtGml2 from cytobank. This is the first reported use case of logtGml2 . I will add the support shortly.

On 12/08/2017 11:45 AM, Dmitry Tebaykin wrote:

Ok, I was able to install the libraries from github. I now have:
flowWorkspace version 3.27.6
openCyto version 1.17.2
flowCore version 1.45.2

My original problem was not being able to import a Gating-ML 2.0 file from Cytobank into R, the error: Error in FUN(X[[i]], ...) :
Don't know how to inverse transformation: logtGml2

I was hoping the updated libraries would be able to deal with that, but the error persists.

It has to do with Makevars.win file. I will update it.

On 02/17/2017 01:01 PM, Dubay, Christopher J wrote:

Anyway, I am trying to install the CytoML R library package from the development thread on GitHub (V1.1.9).

I am getting the attached error message. Somehow have I not installed Google protoc?

Can you please help me get past this error?

The most recent binary for Windows on Biocoductor, that I can see, is CytoML 1.0.1. (there is a 1.1.9 binary for MacOS).

Do you have a more recent Binary for Windows (that would also solve my issue, I think).

Thanks in advance for your time responding to this query...

Christopher

Christopher Dubay

Biomedical Informatics Program Manager

Laboratory of Molecular and Tumor Immunology

Earle A. Chiles Research Institute

Providence Cancer Center

Hi,

I am trying to use the FlowWorkspace package to extract my manual gating set from a FlowJo v10.1 workspace and export that as a GatingML file using "GatingSet2Cytobank". However this fails repeatedly. Here is my code and the console output:

library(CytoML)
library(flowWorkspace)
setwd("/Users/fklemm/Documents/FCM_analysis")
wsp <- "/Users/fklemm/Documents/FCM_analysis/input/csv_flowjo/GBM_46/gbm_46.wsp"
ws <- openWorkspace(wsp)
gs <- parseWorkspace(ws, name=1)
Parsing 1 samples
windows version of flowJo workspace recognized.
version X
Creating ncdfFlowSet...
All FCS files have the same following channels:
FSC-A
FSC-H
FSC-W
SSC-A
SSC-H
SSC-W
FITC-A
BV421-A
Alexa Fluor 700-A
PE-A
BV786-A
APC-A
BV711-A
BV605-A
PerCP-Cy5-5-A
PE-Cy7-A
BV650-A
405nm-585/42-A
PE-Texas Red-A
PE-Cy5-5-A
APC-Cy7-A
Time
done!
loading data: /Users/fklemm/Documents/FCM_analysis/input/csv_flowjo/GBM_46/LAU_GBM_46_bulk_full_panel_002.fcs
Compensating
gating ...
write LAU_GBM_46_bulk_full_panel_002.fcs_2175523 to empty cdf slot...
done!
plot(gs[[1]])
gh <- gs[[1]]
GatingSet2cytobank(gs, "./input/csv_flowjo/GBM_46/gbm_46_gs.xml")
Error in get(y, obj@indices) :
object 'LAU_GBM_46_bulk_full_panel_002.fcs' not found
In addition: Warning message:
In GatingSet2cytobank(gs, "./input/csv_flowjo/GBM_46/gbm_46_gs.xml") :
With 'cytobank.default.scale' set to 'TRUE', data and gates will be re-transformed with cytobank's default scaling settings, which may affect how gates look like.`

Here is my session info:
`> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.6 (El Capitan)

locale:
[1] C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] flowWorkspace_3.20.2 ncdfFlow_2.20.0 BH_1.62.0-1
[4] RcppArmadillo_0.7.600.1.0 CytoML_1.0.0 ggplot2_2.2.0
[7] Rtsne_0.11 FlowSOM_1.6.0 igraph_1.0.1
[10] flowCore_1.40.3

loaded via a namespace (and not attached):
[1] Biobase_2.34.0 splines_3.3.1 jsonlite_1.1
[4] ConsensusClusterPlus_1.38.0 R.utils_2.5.0 gtools_3.5.0
[7] shiny_0.14.2 assertthat_0.1 stats4_3.3.1
[10] latticeExtra_0.6-28 RBGL_1.50.0 robustbase_0.92-7
[13] lattice_0.20-34 quantreg_5.29 RUnit_0.4.31
[16] digest_0.6.10 RColorBrewer_1.1-2 colorspace_1.3-2
[19] htmltools_0.3.5 httpuv_1.3.3 Matrix_1.2-7.1
[22] R.oo_1.21.0 plyr_1.8.4 multicool_0.1-10
[25] pcaPP_1.9-61 XML_3.98-1.5 misc3d_0.8-4
[28] tsne_0.1-3 SparseM_1.74 fda_2.4.4
[31] zlibbioc_1.20.0 xtable_1.8-2 corpcor_1.6.8
[34] mvtnorm_1.0-5 scales_0.4.1 MatrixModels_0.4-1
[37] openCyto_1.12.1 flowStats_3.32.0 tibble_1.2
[40] flowViz_1.38.0 BiocGenerics_0.20.0 hexbin_1.27.1
[43] lazyeval_0.2.0 mnormt_1.5-5 magrittr_1.5
[46] IDPmisc_1.1.17 mime_0.5 mcmc_0.9-4
[49] ks_1.10.4 R.methodsS3_1.7.1 MASS_7.3-45
[52] graph_1.52.0 tools_3.3.1 data.table_1.10.0
[55] flowClust_3.12.1 matrixStats_0.51.0 stringr_1.1.0
[58] MCMCpack_1.3-8 munsell_0.4.3 cluster_2.0.5
[61] flowUtils_1.38.0 grid_3.3.1 htmlwidgets_0.8
[64] base64enc_0.1-3 codetools_0.2-15 gtable_0.2.0
[67] DBI_0.5-1 rrcov_1.4-3 R6_2.2.0
[70] gridExtra_2.2.1 knitr_1.15.1 dplyr_0.5.0
[73] clue_0.3-52 KernSmooth_2.23-15 Rgraphviz_2.18.0
[76] stringi_1.1.2 parallel_3.3.1 Rcpp_0.12.8
[79] rgl_0.96.0 DEoptimR_1.0-8 coda_0.19-1

`

I ultimately want to use the GatingML file within FlowSOM because the xml provided by FlowJo uses cryptic IDs for the gates that makes picking populations tricky.

Thanks a lot, Florian.

Here is the preliminary parsing result, the count difference is probably due to the incorrect hyperlog transformation, which is yet to be done.

> stats
    flowCore.freq   xml.freq     xml.count  flowCore.count          node
 1:    1.00000000  1.00000000       100000         119531          root
 2:    0.77520476  0.77532000        77532          92661        debris
 3:    0.86631916  0.86676469        67202          80274      singlets
 4:    0.63539876  0.63457933        42645          51006           CD3
 5:    0.26053798  0.29185133        12446          13289           CD8
 6:    0.01033212  0.01031774          440            527           DPT
 7:    0.02513430  0.04087232         1743           1282            DN
 8:    0.29933733  0.65695861        28016          15268           CD4
 9:    0.32473146  0.37093090        10392           4958           45+
10:    0.05730941  0.04865077         1363            875 activated CD4

I am trying to compare values from FCS files after importing gates from Cytobank (which applies asinhtGml2 transform) to a static value (Example: 100). Applying the asinhtGml2 to 100 doesn't seem to work. I found the transform(..., inverse = TRUE) option in the

install_github("RGLab/flowWorkspace", ref="trunk")
install_github("RGLab/openCyto", ref="trunk")

versions, but after updating my GatingML file was not recognized as a valid XML and read.flowSet failed to import the FCS files. Is there a plan for updating the bioconductor versions of these libraries to more recent stable versions? If not, what steps should I take to reverse the data transformations after applying gates?

Mike,

I have done some further investigation with respect to the notorious "failed to import" message in Cytobank, that we have been getting more recently when trying to import the GatingML from flow data into Cytobank. To refresh, when I make gates in FJ for flow cytometry files, use CytoML to make a GatingML file, and pass said GatingML file into Cytobank, I get the notorious message we have discussed on other threads:

I have found two things that may be related.

The first is that this "failed to import" message only occurs with markers that are not linear by default (eg not scatter). Below is an example from polygon gate.

The second is that older versions of CytoML produce xml files that still successfully import into Cytobank. In the following example, the older version of CytoML is used on top, and the latest version of CytoML is used on the bottom.

This suggests that a change to CytoML and the related packages could be leading to this problem. There could be a number of reasons for the error, of course (including the older version of R, etc), but a closer look at the xml file reveals that the coordinates within the xml generated by the older version of CytoML has a (logicle?) transformation.

This is interesting, given my previous observation that the non-linear markers are the ones that are affected by this import error. So my question: is it possible that doing this "coordinate transform" for the new xml file could lead to the files being successfully read into Cytobank again? In a separate email, I'll send you the relevant wsp file and xml files.

Thanks for your help!

When the scale settings in flowjo are adjusted (before or after drawing the gate) from a default of width argument = -100, the gate coordinates end up incorrect after transfer to Cytobank. This is to be contrasted with successful transfers observed when the scale settings in flowjo are left at a default width argument of -100.

example:

Seems that a conversion calculation to go from biex to arcsinh isn't accounting for the possibility of a variable width argument? Also it's unclear as to why such a conversion is needed in the first place, since gate coordinates appear to be stored as raw / untransformed coordinates inside the flowjo WSP and could be passed on directly toward Cytobank from this state. Thoughts?

Repro steps

1. Choose a couple channels within the dataset and change the biexponential width argument from -100 to something different such that a meaningful change in the data display is seen.
2. Draw a few gates on the channels for which you changed the scale settings.
3. Transfer to Cytobank and assess accuracy. Dramatic differences can be seen.

Hi, CytoML fails to build in:
http://bioconductor.org/checkResults/devel/bioc-LATEST/CytoML/malbec1-install.html
with a

Error: class "workFlow" is not exported by 'namespace:flowCore'

which also blocks the Docker container devel_cytometry2
https://github.com/Bioconductor/bioc_docker#maintained-by-rglab-wjiang2fredhutchorg

Yours,
Steffen

Gates could not be transferred with the CytoML package when CyTOF files were being used. Errors related to compensation popped out.

GatingSet2cytobank(gs, x.xml)
Error in initialize(value, …) : object ‘compId’ not found

It’s possible that this is because there is no flowjo compensation in these situations. An empty placeholder compensation matrix (spillover matrix) was inserted in one such file and the errors persisted. A natural next step would be to try inserting a compensation matrix with actual values in it to see if the errors persist. This hasn't been attempted yet.

This file (Marrow1_Mature CD8+ T_cells.fcs.zip) is a test file taken from this Cytobank experiment.

This is the first attempt to write a GatingSet to winlist protocol file.
I noticed that there are a lot of stuff in xml that are irrelevant to the gating but somehow winlist software is pretty picky on the input xml so I wasn't able to reduce it into a minimal subset as I did for Cytobank and flowJo.
Therore, I was trying to use the existing .wlx file generated by winlist as my template and make changes to it by removing and adding xml elements to reflect my new cascade gating stored in GatingSet.
However, the new winlist protocol crashes winlist without getting detailed error information.
here is the FCS data

The missing actual calibration table values currently failed the transformation parsing for some mac workspace.

mac version of flowJo workspace recognized.

 *** caught segfault ***

address 0x7fffffff8, cause 'memory not mapped'

Since we still have the other three major biexp parameters (i.e. decade, neg, width) available from xml we can try to compute the calibration table from these assuming maxValue is always 262144 (which is the case as far as I've seen).

When interchanging gating analysis between openCyto and other vendor software (e.g. cytobank, flowJo), CytoML needs to deal with transformation conversions between flowCore::logicle and flowJo biexp back and forth. Based on the current coercion formula

flowJo_to_logicle <- function (maxValue = 262144, widthBasis = -100, pos = 4.5, neg = 0) 
{ 
  myTrans <- logicleGml2_trans(M = pos
                               , T = maxValue
                               , A = neg
                               , W = log10(-widthBasis) / 2)
  myTrans$transform
}

The conversion has been indirectly validated by the gating results (flowJo-->openCyto-->cytobank) .
But it would actually helpful to see if we can compare the transformed data directly.
So here is the test

fj.biexp <- flowJoTrans(maxValue = 262144, widthBasis = -100, pos = 4.5, neg = 0, channelRange = 4096)
fc.biexp <- flowJo_to_logicle(maxValue = 262144, widthBasis = -100, pos = 4.5, neg = 0)  
plot(x = fj.biexp(1:1e5), y = fc.biexp(1:1e5))

Looks pretty linear which indicates the extra channelRange from flowJo is simply a linear-scaling factor. to verify

df <- data.frame(x = fj.biexp(1:1e5), y = fc.biexp(1:1e5))
res <- glm(x~y, data = df)
res <- summary(res)
res$coefficients
   Estimate   Std. Error     t value Pr(>|t|)
(Intercept)   -0.8156886 0.0007357149   -1108.702        0
y           4096.0871602 0.0009049816 4526155.072        0

all.equal(fj.biexp(1:1e5), fc.biexp(1:1e5) * 4096)
[1] "Mean relative difference: 0.0002257089"

However, flowJo seems to be more robust in terms of self-correcting the invalid input whereas flowCore is sensitive and easy to error out

> fj.biexp <- flowJoTrans(maxValue = 262144, widthBasis = -10, pos = 4.8, neg = 100, channelRange = 1024)
> fc.biexp <- flowJo_to_logicle(maxValue = 262144, widthBasis = -10, pos = 4.8, neg = 100)
> plot(x = fj.biexp(1:1e5), y = fc.biexp(1:1e5))
Error in flowCore:::logicle_transform(as.double(x), as.double(T), as.double(W),  : 
  Logicle Exception: IllegalParameter: A is too large

On 06/19/2017 04:23 PM, Chris Ciccolella wrote:

Maybe you can double check this but we can't get it to work. This is when you have the same gate but in a slightly different location between samples. This happens frequently between donors or between acquisition collection dates or instruments.

The spider gates (also quadGates) are parsed from flowJo as individual rectangeGates, which means GatingSet doesn't really have information about the relationship of these quadrants once they are parsed.
For quadGate, I am currently re-associating them by checking the dimension and parent as well as the coordinates. This process is a bit heuristic and I am not sure how to accurately extend it to curly quad or spider gate.
In order to tackle this in a cleaner way, we may have to go back to the cytolib and store the quad information somewhere, which will take some efforts to implement.
Before we make a decision on this, from you perspective, @tjburns08, how crucial is it to have them restored?

@jspidlen, Does flowJo xml have a general way to annotate the bool or integer event indices representing the cell memberships produced by the clustering algorithm? We'd like to run some in-house clustering method and store the cluster membership or labels in GatingSet and then export to flowJo.
I know flowJo supports several specific plugins. Are these population nodes generated by these plugins only meaningful for the existing plugins?(Thus not suitable for storing results from other clustering methods)

Hello,
I am trying to export my gates to flowJo, and I have couple of questions.
My automated gating pipeline is based on estimateLogicle transformation, and it seems likes that it's unsupported for CytoML. So

1. Do I need to change it to flowJo_biexp_trans in order to be able to export and open it in flowJo or there's a way to translate logicle to what flowJo understands? Cause that would change my gating.
   I even tried to pass estimateLogicle parameters to flowJo_biexp_trans(), but I get an error of invalid zeroChan (WidthBasis). I f I only change channelRange to be same as the range in estimateLogicle, it works, but I'd prefer a better (cleaner) way.
2. Also when I use :
   gs <- transform(gs, trans.list) it seems like that it transforms the data on all nodes, does it keep the raw one as well?

These might be basic questions for you, but I'd appreciate it if you can help me.

Typically sample group is parsed from xpath /bdfacs/experiment/specimen as shown below

But we came across the use case using the different xpath with tray node representing sample group

this causes the failure of sample group parsing

> getSampleGroups(ws)
data frame with 0 columns and 0 rows

We may consider exporting flowJo workspace instead of standard gatingML files due to the lack of support of the latter from flowJo.
We've been down that path before, @gfinak , do you recall why it was halted?
@jspidlen, since you've implemented workflow2FlowJo in flowUtils, I'd like have your input on this as well.

Hi Mike,

I would like to manually set xlim and ylim for each channel in a gating set. Then export it as a flowJo workspace. Therefore, when I open the .wsp in flowJo I would like to see the exact same scaling. I have attached a skeleton of my code to give you an idea of what I have attempted. (There were many things calculated that I left out.) I was hoping that the gs_test@axis would be transferred into flowJo, but it isn't the case. Is there something like that which will allow me to have predefined axes in flowJo? I have attached three figures for the data/instrument/custom range part. I also attached a figure of the flowJo scaling.

Thank you,
Justin

fs <- fs_Org[keep.indices[1:2]]

suppressMessages(gs_test <- GatingSet(fs))
gs_test <- compensate(gs_test, M1); suppressMessages(recompute(gs_test))
suppressMessages(nodeMargins  <- add(gs_test, Marg.RG[1:2],     parent="root"));      suppressMessages(recompute(gs_test))
suppressMessages(nodeScatter  <- add(gs_test, scatter.box.PG,   parent="Margins"));   suppressMessages(recompute(gs_test))
suppressMessages(nodeSinglets <- add(gs_test, singlets.PG[1:2], parent="NonDebris")); suppressMessages(recompute(gs_test))
suppressMessages(gs_test <- transform(gs_test, transList)); suppressMessages(recompute(gs_test))

# Just for plotting in R (see attached for figures)
flowWorkspace.par.set("plotGate",list(xlim = "data"))
plotGate(gs_test[[1]], arrange.main="Data")
flowWorkspace.par.set("plotGate",list(xlim = "instrument"))
plotGate(gs_test[[1]], arrange.main="Instrument")
flowWorkspace.par.set("plotGate",list(xlim = c(0,15000000)))
plotGate(gs_test[[1]], arrange.main="Custom Range")

GatingSet2flowJo(gs_test, output.path)

gs_test2 <- gs_test

# part that I hope would manually change the flowJo axes
limits <- list(c(0,5000000), c(0,6000000), c(0,7000000))
names(limits) <- (gs_test2@data[[1]]@parameters@data$name)[1:3]
gs_test2@axis <- limits

# This ended up being exactly the same as the original WorkSpace
GatingSet2flowJo(gs_test2, gsub("WorkSpace", "WorkSpace2",output.path))

In the example above "Viable cells" and "No beads" populations are defined by the same 3 gates {singlets, viable cells, no beads}. This causes an error in cytobank2GatingSet():
Error in addNode(thisPopId, g) : node(s) already in graph: ‘Gate_9033841_dmlhYmxlIGNlbGxz’

In this particular example this was a user mistake ("Viable cells" should only include {singlets, viable cells}). However, there is conceivably cases where we may want to support duplicate populations with different names, so I'm filing this issue to have a record of this behavior.

When importing gatingML files, in constructTree, special characters in gate names (popName, thisGateName) are replaced with : when the node is added through addGate but this change is not reflected in nodeData, which leads to errors when applying the parsed tree to data.

I used the following temporary fix

gsub('\/',':',popName)

But without knowing exactly which characters are replaced by nodeData it is difficult to come up with the correct regex pattern. I'm happy to submit a pull request if you can provide me with this information.

Tried uploading gatingML file exported from cytobank using the "read.gatingML.cytobank" function.
Received this error:
Error in orig_params[ind] : invalid subscript type 'list' uploading gatingML from cytobank
Also tried uploading gatingML using "cytobank2GatingSet" and received same error.
Has anyone seen this before?

After import into Cytobank, quad gates from flowjo are not quad gates but instead are 4 rectangular gates. The gates are huge and their edges extend very far off axis. The statistics labels are also very far off the plot and thus not visible. In order to see the labels the gate must be dragged and resized many times, which is painful.

This issue affects other quad types in FJ such as curly quad and quad-disconnected-center. It doesn't seem to affect spider quad as badly (for this quad type, the edges extend only marginally off axis).

Should be fairly simple to adjust CytoML to not make the rectangles so large. They just need to slightly clear the scale maximum. This ensures that the gates capture all data points beyond the scale maximum, since these data points are binned at the scale maximum in Cytobank. This strategy may solve the problem with the labels without any label-specific logic, since I believe label position is set automatically based on general gate characteristics.

After export gating set to flowjo workspace GatingSet2flowJo, the quadrant gate is parsed by FlowJo as four independent rectangular gates. I compared the xml specification between CytoML and FlowJo:

The difference is as below:

Quadrant gate in FlowJo:

<gating:RectangleGate eventsInside="1" annoOffsetX="0" annoOffsetY="0" tint="#000000" isTinted="0" lineWeight="Normal" userDefined="1" percentX="0" percentY="0" >
<gating:dimension gating:max="86.71224060277042" >
<data-type:fcs-dimension data-type:name="Comp-PE-Texas Red-A" />
</gating:dimension>
<gating:dimension gating:min="546.4788302000138" >
<data-type:fcs-dimension data-type:name="Comp-PE-Cy7-A" />
</gating:dimension>
</gating:RectangleGate>

Quadrant gate in CytoML (difference highlighted):

<gating:RectangleGate eventsInside="1" annoOffsetX="0" annoOffsetY="0" tint="#000000" isTinted="0" lineWeight="Normal" userDefined="1" percentX="0" percentY="0" >
<gating:dimension gating:min="38.9175232999206" gating:max="2.147483647E9" >
<data-type:fcs-dimension data-type:name="Comp-PE-Texas Red-A" />
</gating:dimension>
<gating:dimension gating:min="-2.147483647E9" gating:max="337.826512315656" >
<data-type:fcs-dimension data-type:name="Comp-PE-Cy7-A" />
</gating:dimension>
</gating:RectangleGate>

Could this be improved? I am not sure whether this is more related to flowWorkspace or flowCore

Thank you.

flowJo expect the gate coordinates to be at raw scale. So transformation is required to be present in GatingSet in order to inverse transform the gates.
You can do the compensation and transformation directly to GatingSet instead of FCS so that they will be all recorded in GatingSet. (See vignette )

There is flowWorkspace::asinhtGml2_trans constructor you can use if you prefer to asinh

On 07/12/2016 07:25 AM, Brian Muchmore wrote:

Hi Mike,

So, I run the following code

flowjofile <- file.path
GatingSet2flowJo(gs, flowjofile)

And get the following error

Error: No transformation is found in GatingSet!

I run an asinh transformation on my original FCS files and then rewrite them as new FCS files before running openCyto. Is it possible to augment the GatingSet2flowJo function to ignore the transformation information (or lack thereof)?

Thanks.

-Brian

Running R CMD check gives NOTEs about undefined global functions / variables / bindings

.preprocessorDiva: no visible binding for global variable ‘specimen’
.preprocessorDiva: no visible binding for global variable
  ‘sampleSelected’
GatingSet2flowJo: no visible global function definition for ‘pData<-’
addGate: no visible binding for global variable ‘id’
addGate: no visible binding for global variable ‘gate_id’
addGate: no visible binding for global variable ‘fcs’
addGate: no visible binding for global variable ‘gate_def’
addGate: no visible binding for global variable ‘name’
compare.counts: no visible binding for global variable ‘population’
compare.counts: no visible binding for global variable ‘parent’
compare.counts: no visible binding for global variable ‘count’
compare.counts: no visible binding for global variable ‘parent_count’
compare.counts: no visible global function definition for ‘.’
compare.counts: no visible binding for global variable ‘fcs_filename’
constructTree: no visible binding for global variable ‘id’
constructTree: no visible binding for global variable ‘name’
export_comp_trans: no visible global function definition for ‘logtGml2’
extend.polygonGate: no visible binding for global variable ‘..dim’
extend.polygonGate : <anonymous>: no visible binding for global
  variable ‘y’
extend.polygonGate : <anonymous>: no visible binding for global
  variable ‘x’
extend.polygonGate: no visible binding for global variable ‘id’
extend.polygonGate: no visible binding for global variable ‘x’
extend.polygonGate: no visible binding for global variable ‘y’
extend.polygonGate: no visible binding for global variable ‘is.smaller’
read.gatingML.cytobank: no visible binding for global variable ‘id’
read.gatingML.cytobank: no visible binding for global variable
  ‘comp_ref’
getTransformations,graphGML : <anonymous>: no visible global function
  definition for ‘logtGml2_trans’
Undefined global functions or variables:
  . ..dim comp_ref count fcs fcs_filename gate_def gate_id id
  is.smaller logtGml2 logtGml2_trans name pData<- parent parent_count
  population sampleSelected specimen x y

These need to be addressed by

1. importing the methods or functions explicitly via roxygen tags and adding package dependencies to DESCRIPTION
2. identifying variables that are not properly defined, they may be bugs.

I used the following command CytoML::GatingSet2flowJo(manual_gs, outFile = "./manual_gs.xml") to generate an .xml file for a gatingset.
The error in the uploaded file seems to be on line 20: The prefix "transforms" for element "transforms:spilloverMatrix" is not bound.
The attached file is with the txt extension (actually is the xml file I'm trying to parse)
manual_gs-sublime2.txt
I am currently using ‘CytoML’ version 1.7.9 within R-3.5.1

Hi all,

I am reading in an xml file created from cytobank with read.gatingML.cytobank() and subsequently create a gatingSet with cytobank2GatingSet(). When I then call getData on this gatingSet, the channels that are used for gating get transformed, whereas other channels stay untransformed.

Is there an easy way to prevent transformation when calling getData()?

Cheers, Felix

Hello, very interested in using CytoML to export my GatingSet for others to use FlowJo for their own exploration of the data. However when I export my GatingSet using GatingSet2flowJo the resulting wsp file is empty, am I supposed to do something special to the GatingSet before it is ready to be exported? Additionally, do I need to put the fcs files in the same folder as the exported wsp file?

Hi MIke,
Thanks for using data.table! We include all Bioconductor packages using data.table when reverse dependency testing.

When v1.11.0 goes to CRAN, the breakage will be as follows :

$ tail CytoML-Ex.Rout
..done!
> ## verify the stats are correct
> statsfile <- system.file("extdata/cytotrol_tcell_cytobank_counts.csv", package = "CytoML")
> dt_merged <- compare.counts(gs, statsfile, id.vars = "population")
Warning in fread(file, stringsAsFactors = FALSE) :
  Stopped early on line 7. Expected 2 fields but found 0. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<FCS Filename,CytoTrol_CytoTrol_1.fcs,CytoTrol_CytoTrol_2.fcs>>
Error in setnames(cytobank_counts, "FCS Filename", "population") : 
  Items of 'old' not found in column names: FCS Filename
Calls: compare.counts -> setnames
Execution halted

Looking at that file :

$ more ~/build/revdeplib/CytoML/extdata/cytotrol_tcell_cytobank_counts.csv
Experiment Name:,cytotrol_tcell
Project Name:,test
Primary Researcher:,Mike Jiang
Compensation used:,File-internal Compensation
Exported at:,December  7 2015 - 11:33 am
Link to experiment:,https://www.cytobank.org/cytobank/experiments/48376

FCS Filename,CytoTrol_CytoTrol_1.fcs,CytoTrol_CytoTrol_2.fcs
not debris,87876,87648
singlets,79845,79641
CD3,53138,50782
CD8,12863,12418
CD4,33654,32237
Q1,419,384
[ ... snip ... ]

fread in v1.11.0 now, by default, reads the first block of contiguous table-like data it can find. Because of those commas in the fields in the banner, it returns that 6x2 table. It warns you that data afterwards has been discarded, too.

 > data.table:::fread("cytotrol_tcell_cytobank_counts.csv")
      Experiment Name:                                      cytotrol_tcell
                <char>                                              <char>
1:       Project Name:                                                test
2: Primary Researcher:                                          Mike Jiang
3:  Compensation used:                          File-internal Compensation
4:        Exported at:                         December  7 2015 - 11:33 am
5: Link to experiment: https://www.cytobank.org/cytobank/experiments/48376
Warning message:
In data.table:::fread("cytotrol_tcell_cytobank_counts.csv") :
  Stopped early on line 7. Expected 2 fields but found 0. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<FCS Filename,CytoTrol_CytoTrol_1.fcs,CytoTrol_CytoTrol_2.fcs>>

The most robust solution is to use the skip="string" feature and choose any unique substring of the columns names you need. In this case: "FCS Filename".

> data.table:::fread("cytotrol_tcell_cytobank_counts.csv", skip="FCS Filename")
    FCS Filename CytoTrol_CytoTrol_1.fcs CytoTrol_CytoTrol_2.fcs
          <char>                   <int>                   <int>
 1:   not debris                   87876                   87648
 2:     singlets                   79845                   79641
 3:          CD3                   53138                   50782
 4:          CD8                   12863                   12418
 5:          CD4                   33654                   32237
 6:           Q1                     419                     384
 7:           Q2                   11429                   10905
 8:           Q3                   17687                   16971
 9:           Q4                    4119                    3977
10:       CD8_Q2                    2332                    2211
11:   CD38 range                    2332                    2211
12:    HLA range                    2316                    2196
>

That change will work will old versions of data.table as well as v1.11.0.
I hope this change is ok to make. Sorry for the inconvenience.
Best, Matt

as the workaround to represent the same node name under different gating path in cytobank. See details below

As far as I can see, 'tailored gate' in Cytobank is restricted to the same type of gate defined at the same dimensions. There isn't such restriction in FlowJo or openCyto, that means technically 'CD4/IL2+' and 'CD8/IL2+' could be entirely two different unrelated gates regardless of the same name 'IL2+'. I think I've seen such use cases in the past.

At least it should cover majority of cases, I hope. We can deal with the exceptions later as we encounter them.

On 02/12/2018 03:40 PM, Preston Ng wrote:

    2.1 No duplication of population name is NOT allowed, even if they have different gating path
    i.e. place the same type of marginal gates (e.g ‘IL2’) under different parents are very common in our analysis, which resulted in the two populations with the common name ‘IL2’ ,but with unique gate path for each (‘CD4/IL2+’ and ‘CD8/IL2+’)

I believe what you're talking about is called Tailored Per Population if I recall? Or are you just talking about creating another population with the same gating path to define that population?

Thanks for your work on this package! I've run into the following issue: the read.gatingML.cytobank appears to fail when I attempt to read in an exported cytobank gatingML file with the following error message: (I'd be happy to share the gatingML file, but I'm not sure how to add attachments to github).

library(flowUtils)
library(CytoML)

file <-  "CytExp_29783_Gates_v5.xml"
gatingML.cytobank <- read.gatingML.cytobank(file)

Error in addGate(gateInfo, flowEnv, g, thisPopId, gateID) :
multiple gates found for Gate_1590224_dmlhYmxl

This gate corresponds to a polygon gate drawn against FSC and Ax700.
On inspecting the gatingML file, there appears to be only one Gate_1590224_dmlhYmxl defined.

<gating:PolygonGate gating:id="Gate_1590224_dmlhYmxl">
    <data-type:custom_info>
      <cytobank>
        <name>viable</name>
        <id>1590224</id>
        <gate_id>4</gate_id>
        <type>PolygonGate</type>
        <version>5</version>
        <compensation_id>3519</compensation_id>
        <fcs_file_id />
        <tailored>false</tailored>
        <tailored_per_population>false</tailored_per_population>
        <tailored_per_population_gateset_id />
        <fcs_file_filename />
        <definition>{"scale":{"x":{"flag":1,"argument":"1","min":1.0,"max":262144.0,"bins":256,"size":256},"y":{"flag":4,"argument":"1500","min":-4000.0,"max":262144.0,"bins":256,"size":256}},"positive":false,"negative":false,"label":[171153.05752893395,1.6602280372211227],"polygon":{"vertices":[[140789.0034748799,1.1618252220891794],[32490.54401542045,1.1618252220891794],[7187.1656370420715,0.48751553102831524],[17308.51698839342,-1.4181422915349964],[44636.165637042075,-1.8579094813572992],[179250.13861001504,-1.3888244788801762],[200504.9764478529,0.04774834120601246],[171153.05752893395,1.3670499106729208]]}}</definition>
      </cytobank>
    </data-type:custom_info>
    <gating:dimension gating:compensation-ref="uncompensated">
      <data-type:fcs-dimension data-type:name="FSC-A" />
    </gating:dimension>
    <gating:dimension gating:compensation-ref="Spill_3519" gating:transformation-ref="Tr_Arcsinh_1500"> 
      <data-type:fcs-dimension data-type:name="Comp_Alexa Fluor 700-A" />
    </gating:dimension>
    <gating:vertex>
      <gating:coordinate data-type:value="140789.0034748799" />
      <gating:coordinate data-type:value="1.1618252220891794" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="32490.54401542045" />
      <gating:coordinate data-type:value="1.1618252220891794" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="7187.1656370420715" />
      <gating:coordinate data-type:value="0.48751553102831524" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="17308.51698839342" />
      <gating:coordinate data-type:value="-1.4181422915349964" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="44636.165637042075" />
      <gating:coordinate data-type:value="-1.8579094813572992" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="179250.13861001504" />
      <gating:coordinate data-type:value="-1.3888244788801762" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="200504.9764478529" />
      <gating:coordinate data-type:value="0.04774834120601246" />
    </gating:vertex>
    <gating:vertex>
      <gating:coordinate data-type:value="171153.05752893395" />
      <gating:coordinate data-type:value="1.3670499106729208" />
    </gating:vertex>
  </gating:PolygonGate>

The flowUtils::read.gatingML function is able to read the gatingML file, and also only finds a single version of the gate in question:

flowEnv <- new.env()
gatingML <- read.gatingML(file, flowEnv)
table(grep("Gate_", names(flowEnv), value = T))

> table(grep("Gate_", names(flowEnv), value = T))
                Gate_1590221_aW50YWN0         Gate_1590222_c2luZ2xlcy1zc2M.         Gate_1590223_c2luZ2xlcy1mc2M.                 Gate_1590224_dmlhYmxl 
                                    1                                     1                                     1                                     1 
            Gate_1590225_YmFyY29kZWQ.                 Gate_1590226_cEJBQkU. Gate_1590227_cEJBQkUtSFJhc0cxMlYrSUwz Gate_1590228_cEJBQkUtSFJhc0cxMlYtSUwz 
                                    1                                     1                                     1                                     1 
Gate_1590229_cE1TQ1YtRUdGUl9MODU4Ug.. 
                                    1 

In my attempt to debug it, I believe the duplication is occurring within the parse.gateInfo() function: as can be seen here:

Browse[2]> gateInfo[,1:7]
                                       id               name gate_id fcs      comp_ref       trans_ref                                                 params
 1:                 Gate_1590221_aW50YWN0             intact       1     uncompensated                                                            SSC-A:FSC-A
 2:         Gate_1590222_c2luZ2xlcy1zc2M.        singles-ssc       2     uncompensated                                                            SSC-H:SSC-W
 3:         Gate_1590223_c2luZ2xlcy1mc2M.        singles-fsc       3     uncompensated                                                            FSC-W:FSC-H
 4:                 Gate_1590224_dmlhYmxl             viable       4     uncompensated Tr_Arcsinh_1500                           FSC-A:Comp_Alexa Fluor 700-A
 5:                 Gate_1590224_dmlhYmxl             viable       4        Spill_3519 Tr_Arcsinh_1500                           FSC-A:Comp_Alexa Fluor 700-A

It appears that the issue is that one of the gating axis is compensated (Ax700) while the other one is not (FSC-A), which the function interprets at there being two versions of the gate drawn on different compensation matrices.

Hello,

I'm setting up a shinyApp fir automated analysis, and fter running the analysis, the results can be downloaded.
Everything will be saved in a temp directory using tempdir(). I was able to save everything except the derived parameters, as they will be saved like "./sample****.EPA.csv" and I'd like to have the option of removing "./" so everything will be saved in a same folder. Is there a work around this?

Thanks in advance,

Mehrnoush

On 05/10/2017 05:33 AM, Chris Ciccolella wrote:

Ah, so, Cytobank does accept the scale settings on a per-gate basis in the gatingml, it just doesn't set the experiment scales to match. The end result is that the scales for the gate itself are set however you like, but you then have to go into the experiment and change them to match the gate. It's a pain, but it's doable. So put another way, there is no reason that the package couldn't have this functionality. To clarify, the package itself currently doesn't support this and that's a known issue?

Regarding scale presets, it actually depends on the cytometer that produced the files. A typical fluorescent preset for a variety of cytometers is what you provided. Cytof, though, has min of -5, max of 12k, and scale argument of 5.

Thanks!
Chris

Sent from mobile

On May 10, 2017, at 2:06 AM, Mike [email protected] wrote:

It won't work unless cytobank can parse the scale information from the gatingML and set the Experiment Scales accordingly. As far as I know from my last try, cytobank presets Arcsinh with scale argument = 150 and range = (-200, 262144.0) and ignores what's actually provided in gatingML.

On 05/08/2017 09:53 PM, Chris Ciccolella wrote:
Hi Mike,

Hope you're doing well. Quick question for you regarding CytoML.

We're having trouble with the cytobank.default.scale argument being set to false. I think setting this argument to false will be critical for proper transfer of gates when any gating happens outside the boundary of Cytobank default scales, which will be reasonably often. Do you have any insight for this error message (attached) and are you certain that the functionality works for you?

Chris

I'd like to see if I can use parseWorkspace to read in a FlowJo v9 workspace and then export it as a FlowJo v10 workspace.

I saw in issue #5 that transformation and compensation needs to be applied to the GatingSet (not the flowSet) in order to be exportable using GatingSet2flowJo.

Is it possible to first run parseWorkspace with execute = FALSE, create a GatingSet from it, and then apply a compensation matrix to the GatingSet? When I try this, I have trouble finding the matrix. It doesn't seem to be stored directly in the FCS files.

The only compensation-related keyword for the Samples (non-Comp) FCS files is:

$FJ_CompMatrixName
[1] "COMPS, tetramer and L/D"

This is the name of the project-specific compensation matrix that was applied in FlowJo 9 prior to gating.
The Compensation FCS files seem to have SPILL keywords but I believe they are the identity matrix...

Thanks for your advice.

Edit: Compensation and gating work fine with execute = TRUE

Moving RGLab/flowWorkspace#10 here.

To summarize, here are parts that have been successfully parsed:

1. the specimen (i.e. group) and tube (i.e. sample) nodes
2. compensation node
3. gate node (include rectangle, interval, polygon region)

We have not been able to parse the transformation correctly since it is not clear where the biexp information are stored in xml. So tt is currently applying the biexp transformation with default parameter settings.

Here is the example run of diva parser

> library(flowWorkspace)
> library(CytoML)
> ws <- openDiva(system.file('extdata/diva/PE_2.xml', package = "CytoML"))
> ws
Diva Workspace Version  Version 6.1.3 
File name:  PE_2.xml 
Workspace is open. 

Groups in Workspace
               specimen samples
1 Compensation Controls       9
2                    PE       4
> gs <- parseWorkspace(ws, name = 2, subset = 1)
Parsing 1 samples
loading data: 124500.fcs
Compensating
transforming ...
parsing gates ...
.done!
> plotGate(gs[[1]])

As shown, the gates in P3 and P4 are incorrect due to the inaccurate biexp settings.

Both GatingSet2flowJo and GatingSet2cytobank generate the psuedo gatingML files, which work pretty well for their own purposes.
However the application described in #13 calls for a genuine gatingML to be used by flowSOM package.
This can be done by leveraging existing modules from GatingSet2cytobank and disable the gate rescaling behavior.

The following methods are not exported by flowWorkspace but are used in CytoML

checking dependencies in R code ... NOTE
Unexported objects imported by ':::' calls:
  ‘flowUtils:::.fuEnv’ ‘flowUtils:::smartTreeParse’
  ‘flowWorkspace:::.cpp_getCompensation’
  ‘flowWorkspace:::.fix_channel_slash’ ‘flowWorkspace:::.groupByTree’
  ‘flowWorkspace:::compute.timestep’ ‘flowWorkspace:::isHidden’
  ‘flowWorkspace:::isNegated’
  See the note in ?`:::` about the use of this operator.

We may want to consider exporting them to eliminate the NOTEs in R CMD check.

On 01/11/2017 01:23 PM, Florian Klemm wrote:

Hi Mike,

As an afterthought to my previous email, I realized that the data I extract with getData from my gating hierarchy results in a flowframe with a range of 206.2144 for most of my fluorescence channels. How does that relate to the coordinates for the gates in the exported xml file. Here all the values are between around -5 and 5? Is there a way that back to the original range while keeping flowJo’s transformations? Using “getTransformations” and then transform didn’t work because the object retrieved by the former was not accepted as a transformlist by the latter.

Best, Florian.

From: Florian Klemm [email protected]
Date: Wednesday 11 January 2017 at 18:05
To: Mike [email protected]
Subject: Re: parsing FlowJo workspace and saving gating set

Hi Mike,

Thanks a lot for fixing the error. Saving the gs as a xml works now for me. Unfortunately, I encountered enother error when setting the scale option to FALSE. Please see the console output below. I want to avoid rescaling the data because then I can’t use the gates on my flowEnv that I need for flowSOM, as their position in relation to the original data will have changes (or am I wrong when assuming this?).

Best regards, Florian.

GatingSet2cytobank(gs, "./input/csv_flowjo/GBM_46/gbm_46_gs.xml", cytobank.default.scale = FALSE)

Error in FUN(X[[i]], ...) :

can't find the transformation function in GatingSet to inverse the range for :PE-A

It would be very useful to be able to import a cytobank experiment into flowCore, including all of the preprocessing steps which are not included in the cytobankGatingML file. This functionality could be similar to the openWorkspace() function in flowWorkspace, but for cytobank instead of flowJo.

The functions implemented in CytoML and flowWorkspace allow for reproducing gating schemes from cytobank, but do not allow for importing the complete workspace: including scales (for non-gated channels), sample tags, compensations (those not included in compensations), and panels.

Ideally, I would like to be able to build a transformer list which reproduces the scales defined in cytobank, rename markers according to panels, and include pData as defined by sample tags.

I could imagine two approaches to implementing this which would both be useful:

1. Importing a locally stored experiment which has been exported from cytobank as an ACS file.
   (This data appears to be included in the cytobank_experiment.yaml included in the ACS file.)
2. Importing an experiment from cytobank directly using the CytobankAPI package.
   (This data can be accessed from a series of API calls to cytobank, which could be bundled together as a single workspace object).

I'd be happy to contribute to implementing these features, but wasn't sure if they'd be more appropriate for flowWorkspace, CytoML, or a separate package all together.

On 09/27/2017 09:39 AM, Tyler Burns wrote:
Mike,

When I try to run CyTOF data through CytoML, I get the following error, upon running GatingSet2Cytobank():

"Error in export_comp_trans(gs, flowEnv, cytobank.default.scale = cytobank.default.scale, :
unsupported trans: flowJo_flog"

I was wondering if you could have a look? I have attached a CyTOF fcs and wsp file (name starts with marrow), a control flow fcs and wsp file that works fine (name starts with flow), and the CytoML-utilizing script I used.

Thanks,

Hi Mike,

I try to use your patch in CytoML package to open and use diva template xml (with openDiva function).

But during the installation, i have an error on flowWorkspace when i install this package from github.
I follow the installation guide on https://github.com/RGLab/CytoML and i receive this error

> install_github("RGLab/flowWorkspace", ref="trunk")
Downloading GitHub repo RGLab/flowWorkspace@trunk
from URL https://api.github.com/repos/RGLab/flowWorkspace/zipball/trunk
Installing flowWorkspace
"C:/PROGRA~1/R/R-33~1.0/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  "C:/Users/qbarbier/AppData/Local/Temp/RtmpauGS6z/devtools1f38c52120a/RGLab-flowWorkspace-0dedd8c" --library="C:/Users/qbarbier/Documents/R/win-library/3.3" --install-tests 

* installing *source* package 'flowWorkspace' ...

   **********************************************
   WARNING: this package has a configure script
         It probably needs manual configuration
   **********************************************


** libs

*** arch - i386
mkdir -p "C:/Users/qbarbier/Documents/R/win-library/3.3/flowWorkspace/include/flowWorkspace"
cp include/* C:/Users/qbarbier/Documents/R/win-library/3.3/flowWorkspace/include/flowWorkspace
cp GatingSet.pb.h C:/Users/qbarbier/Documents/R/win-library/3.3/flowWorkspace/include #GatingSet.pb.h is autogenerated by protoc and thus kept at the same folder as GatingSet.proto
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=c++0x -I"C:/PROGRA~1/R/R-33~1.0/include" -DNDEBUG -DROUT -I//i386/include/libxml2 -I//i386/include -DLIBXML_STATIC   -I"C:/Users/qbarbier/Documents/R/win-library/3.3/Rcpp/include" -I"C:/Users/qbarbier/Documents/R/win-library/3.3/BH/include" -I"d:/Compiler/gcc-4.9.3/local330/include"     -O2 -Wall  -mtune=core2 -c GatingHierarchy.cpp -o GatingHierarchy.o
In file included from include/calibrationTable.hpp:20:0,
                 from include/transformation.hpp:23,
                 from include/gate.hpp:15,
                 from include/POPINDICES.hpp:15,
                 from include/nodeProperties.hpp:11,
                 from include/populationTree.hpp:11,
                 from include/GatingHierarchy.hpp:13,
                 from GatingHierarchy.cpp:8:
include/../GatingSet.pb.h:9:42: fatal error: google/protobuf/stubs/common.h: No such file or directory
 #include <google/protobuf/stubs/common.h>
                                          ^
compilation terminated.
make: *** [GatingHierarchy.o] Error 1
Warning: running command 'make -f "Makevars.win" -f "C:/PROGRA~1/R/R-33~1.0/etc/i386/Makeconf" -f "C:/PROGRA~1/R/R-33~1.0/share/make/winshlib.mk" CXX='$(CXX1X) $(CXX1XSTD)' CXXFLAGS='$(CXX1XFLAGS)' CXXPICFLAGS='$(CXX1XPICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX1XLDFLAGS)' SHLIB_LD='$(SHLIB_CXX1XLD)' SHLIB="flowWorkspace.dll" OBJECTS="GatingHierarchy.o GatingSet.o GatingSet.pb.o POPINDICES.o R_API.o R_GatingHierarchy.o R_GatingSet.o RcppExports.o calibrationTable.o compensation.o delimitedMessage.o flowData.o flowJoWorkspace.o gate.o getSingleCellExpression.o global.o macFlowJoWorkspace.o nodeProperties.o spline.o transformation.o winFlowJoWorkspace.o workspace.o wsNode.o"' had status 2
ERROR: compilation failed for package 'flowWorkspace'
* removing 'C:/Users/qbarbier/Documents/R/win-library/3.3/flowWorkspace'
* restoring previous 'C:/Users/qbarbier/Documents/R/win-library/3.3/flowWorkspace'
Error: Command failed (1)

I don't know if the problem comes from my R version or my flow cytometry library,
i copy my session information

R version 3.3.0 (2016-05-03)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] devtools_1.12.0      BiocInstaller_1.24.0

loaded via a namespace (and not attached):
[1] httr_1.2.1    R6_2.2.0      tools_3.3.0   withr_1.0.2   curl_2.3      memoise_1.0.0 knitr_1.15.1  git2r_0.18.0  digest_0.6.12

Do you think the problem can be comes from R 3.3.0 ?

Thank a lot for your help and your work,
Quentin Barbier

R version: 3.4.2 on Mac OS X 10.12.6 Sierra.

The older version (0.5.40 of 2011-03-04) of flowWorkspace works, but it does not have the logtGml2 that Cytobank uses these days. As a result I cannot import the Cytobank gates into R.

When following the installation steps:
biocLite("openCyto")
install_github("RGLab/flowWorkspace", ref="trunk")
install_github("RGLab/openCyto", ref="trunk")

openCyto installs properly, but flowWorkspace installation fails with the following text:

The error message:
R_API.cpp:42:21: error: non-const lvalue reference to type 'GatingHierarchy' cannot bind to a temporary of type 'GatingHierarchy *'
GatingHierarchy & gh = gsPtr->getGatingHierarchy(sn);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R_API.cpp:44:68: error: member reference type 'GatingHierarchy *' is a pointer; did you mean to use '->'?
StringVec allNodes = gsPtr->getGatingHierarchy(sampleNames.at(i)).getPopPaths(REGULAR, isFullPath, true);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
->
R_API.cpp:101:14: error: no viable conversion from 'valarray' to 'vector'
ARRAY_TYPE tmp = cal.getX();
^ ~~~~~~~~~~
/Library/Developer/CommandLineTools/usr/include/c++/v1/vector:530:5: note: candidate constructor not viable: no known conversion from 'valarray' to 'initializer_list<value_type>' (aka 'initializer_list') for 1st argument
vector(initializer_list<value_type> __il);
^
/Library/Developer/CommandLineTools/usr/include/c++/v1/vector:542:5: note: candidate constructor not viable: no known conversion from 'valarray' to 'const std::__1::vector<double, std::__1::allocator > &' for 1st argument
vector(const vector& __x);
^
/Library/Developer/CommandLineTools/usr/include/c++/v1/vector:548:5: note: candidate constructor not viable: no known conversion from 'valarray' to 'std::__1::vector<double, std::__1::allocator > &&' for 1st argument
vector(vector&& __x)
^
R_API.cpp:177:21: error: non-const lvalue reference to type 'GatingHierarchy' cannot bind to a temporary of type 'GatingHierarchy *'
GatingHierarchy & gh = gsPtr->getGatingHierarchy(*it);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11 warnings and 4 errors generated.
make: *** [R_API.o] Error 1
ERROR: compilation failed for package ‘flowWorkspace’

• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/flowWorkspace’
• restoring previous ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/flowWorkspace’
  Installation failed: Command failed (1)

The issue is reported by @cjungR in #25 (comment)
CytoML writes the scale info for this gate in json definition

{"scale":{
"x":{"flag":4,"argument":"5","min":-5,"max":12000,"bins":256,"size":256},
"y":{"flag":4,"argument":"5","min":-5,"max":12000,"bins":256,"size":256}
}}

The label field that defines the stats position in the plot is optional and thus is currently omitted and has working fine for all close-ended gates until we encountered this gate generated by openCyto as open-ended rectangleGate to capture the cells in UR quadrant.

In ggcyto, we superseded the gate range with data range when the open-ended situation occurs, here is the code https://github.com/RGLab/ggcyto/blob/trunk/R/stat_position.R#L76-L79

The solution is either Cytobank needs to fix it or CytoML has to explicitly provide the label info in json.
However, it is not clear how the label coordinates are scaled. Here is the manual gate drawn and output from cytobank

{"scale":{
"x":{"flag":4,"argument":"5","min":-5,"max":12000,"bins":256,"size":256},
"y":{"flag":4,"argument":"5","min":-5,"max":12000,"bins":256,"size":256},
"label":[5.571834733508659,7.968586637265983]
}}

And it seems to be different from the data scale.

libxml2 complains about the xml generated by GatingSet2flowJo on windows

C:/Users/biocbuild/bbs-3.4-bioc/tmpdir/Rtmp8cx1wJ/file1eac74a944f.wsp:1: parser error : Unsupported encoding ISO8859-1
  <?xml version="1.0" encoding="ISO8859-1"?><Workspace version="20.0" xmlns:xsi="h

Looks like it doesn't like the default format returned by R

> localeToCharset()
[1] "ISO8859-1"

The fix would be insert the dash manually . i.e. ISO-8859-1

Hello,

I have some CyTOF files that I have gated using opencyto, and I'd like to export the GatingSet into cytobank with CytoML but I am encountering this error:

> outFile <- tempfile(fileext = ".xml")
> GatingSet2cytobank(gs, outFile)
Error in FUN(X[[i]], ...) : unsupported transform: unitytransform
In addition: Warning message:
In GatingSet2cytobank(gs, outFile) :
  With 'cytobank.default.scale' set to 'TRUE', data and gates will be re-transformed with cytobank's default scaling settings, which may affect how gates look like.

I did transform the .fcs files with a GatingML file exported from cytobank. Could this be the reason why? Is there a workaround for this issue? Thank you for your help.

CytoML is now availble on BioC development branch
http://bioconductor.org/packages/3.4/bioc/html/CytoML.html
So the easiest way for windows users is to install the pre-built version through biocLite, which will take care of all the library dependencies and compiling details.

On 07/09/2016 06:04 PM, Sean Garrity wrote:

Dear Mike,
I'm very interested in testing CytoML. To do so I've been trying to build the most recent flowWorkspace from the RGlab github but am having an issue. The build fails with "collect2.exe: error: ld returned 1 exit status" after listing out a bunch of undefined references (see partial output below).

I don't think this is an RTools issue as I can successfully compile other packages (like openCyto). I also was careful to get the libxml2 and protobuf libraries (as binaries from RGlab) in place and mapped to the environment variables LIB_XML2 and LIB_PROTOBUF, respectively per the install instructions. I'm hoping there's something else obvious that I'm missing. Can I ask you for your thoughts?
Kind regards,
Sean

.....
/home/wjiang2/Downloads/mingw32/src/../../protobuf/src/google/protobuf/stubs/substitute.cc:59: undefined reference to _Unwind_SjLj_Register' /home/wjiang2/Downloads/mingw32/src/../../protobuf/src/google/protobuf/stubs/substitute.cc:62: undefined reference to_Unwind_SjLj_Unregister'
/home/wjiang2/Downloads/mingw32/src/../../protobuf/src/google/protobuf/stubs/substitute.cc:63: undefined reference to `_Unwind_SjLj_Resume'
collect2.exe: error: ld returned 1 exit status
no DLL was created
ERROR: compilation failed for package 'flowWorkspace'

• removing 'C:/Users/sjgarrity/R-Portable/App/R-Portable/library/flowWorkspace'
• restoring previous 'C:/Users/sjgarrity/R-Portable/App/R-Portable/library/flowWorkspace'
  Error: Command failed (1)

Finally, This use case breaks my assumption that the gates we are dealing with are always convex .
Here is the error reported by @ptvan

 gs2 <- cytobank2GatingSet(xml2, fcs2)
singlets
viable cells
no beads
Error in extend.polygonGate(this_gate, bound = bound) : 
  Unsupported number of intersected points with min boundary on x axis: 4

And here is the no beads gate

here is the zoomed-in view

If this concavity was just an accident not really meant by the user, I wonder if we can simply turn it into the convex by chull

which will be just one-line of code and hopefully addresses the issue without compromising the accuracy too much.