Git Product home page Git Product logo

egasub's Introduction

Codacy Badge Build Status Coverage Status

EGASUB - ICGC EGA Submission CLI

EGASUB is a command line tool assists ICGC members submitting their NGS sequence data to EGA repository.

Installation

Install pipsi first (if not installed yet)

# install pipsi
curl https://raw.githubusercontent.com/mitsuhiko/pipsi/master/get-pipsi.py | python

# modify PATH to find pipsi, you may want to add this to the '.bashrc' file
export PATH="~/.local/bin:$PATH"

Install latest egasub release

pipsi install egasub

Note: to install a specific version of egasub, e.g., 0.1.0rc11 do this:

pipsi install egasub==0.1.0rc11

Upgrade to latest release from an old one

pipsi upgrade egasub

If you'd like to play with the latest development version (not recommended for production use), install it from source as below.

# clone the source code
git clone https://github.com/icgc-dcc/egasub.git

# run tests
cd egasub
python setup.py test

# install egasub
pipsi install .

Run

Once installed, you can invoke the tool as follow. This will display information about how to use sub-commands and their options.

egasub

SOP for ICGC submitters

Detailed SOP for ICGC submitters is availabe here.

Support

Should you need assistance, please contact ICGC DCC at [email protected].

egasub's People

Contributors

andricdu avatar baminou avatar chen2x avatar codacy-badger avatar junjun-zhang avatar

Stargazers

avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

navarrabiomed chen2x

egasub's Issues

specify location of files on EGA staging server

This is a feature request.

Organization of the submission locally is dependent on a defined directory structure that reflects the sample/lane. The current too demands that this structure is mirrored on the remote filesystem.

There is some convenience to uploading multiple files to a common folder on the staging server, organizing by different projects, subprojects and types of data. It would be good if in the run.yaml or analysis.yaml if i could explictily indicate the location of the files on the staging server, instead of relying on them being in an identical structure as on the local filesystem.

Allow previously submitted samples to bypass local validation

For the analysis.xml, if I provide a alias that already exists for the sample, the sample information won't be submitted. An INFO message is returned indicating that the alias already exists
eg
sample with alias 'PCSI_0001_Pa_P' already exists in 'SUBMITTED' status, not submitting.

However, there is still an attempt to do local validation on the sample metadata, and this fails if certain fields are not provided.

I'd recommend including a field in the yaml where one can explicitly indicate that the sample already exists at EGA. This should result in local validation not being required for the sample metadata.

This may require follow up during submission to EGA, to ensure that the Alias does actually exist.

EGASub does not allow dot character in submission directory name

Current naming convention for submission directories is {sample_alias}{.lane_label|optional}
See here for more details

The dot character allows data submitters to specify the lane label for unaligned data, however the current implementation gives an error message if a dot character is used. Here is an example of using a submission directory called "tumour_3a.lane1":

Command:
egasub dry_run tumour_3a.lane1

Error Messages:

[ERROR] Local validation error for submission dir 'tumour_3a.lane1': {'object_alias': 'tumour_3a.lane1', 'field': 'alias', 'object_type': 'sample', 'error': "Invalid value 'tumour_3a.lane1'. Sample alias must be set and match the 'alias' portion in the submission directory 'tumour_3a'."}
[ERROR] Local validation error for submission dir 'tumour_3a.lane1': {'object_alias': 'tumour_3a.lane1', 'field': 'alias', 'object_type': 'sample', 'error': "Invalid value 'tumour_3a.lane1'. Sample alias must only contain [a-zA-Z0-9], underscore (_) or dash (-)."}

Analysis objects, submitting multiple bam files

I've discussed with JunJun the possibility of submitting multiple bam files as analysis objects. He indicated that i should be able to provide lane extensions to each sample folder, with one bam file in each
{sample_alias}.{lane_label}
similar to what can be done for unaligned data.

I've tried this and ran egasub dry_run
Error : Skip 'PCSI_0092_Pa_P.05252012' as it appears to be not a well-formed submission directory. Error: Submission directory should be named as , sample alias may only contain letter, digit, underscore (_) or dash (-)

Allow additional information be provided as tag-value pairs via 'attributes'

EGA allows additional information to be provided when registering objects, outside of what is indicated in the yaml templates for analysis and run objects and explained in the SOP.

for example, I've added the following lines to my analysis.yaml, to provide some detail on how the analysis was done

analysis_attributes:
analysis_attribute:
tag: aligner
value: bwa
analysis_attribute:
tag: aligner_ver
value: 0.6.2
analysis_attribute:
tag: mark_duplicates
value: picard
analysis_attribute:
tag: mark_duplicates_ver
value: 1.90

This was info i had provided previously when using XML through the rest server

It would be good to indicate that this is possile in the SOP.

error running egasub : can not get ICGC ID

Hi Jun Jun,
I've started using egasub again.
I've managed to dry_run, submit a number of samples, but others are all failing with the following error.

ERROR] Failed processing 'PCSI_0101_Si_R', can not get ICGC ID, error: Failed calling ICGC ID serivce with: https://id.icgc.org/sample/id?submittedProjectId=PACA-CA&submittedSampleId=PCSI_0101_Si_R&create=true. Server response: <title>Apache Tomcat/8.5.5 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style>

HTTP Status 500 - Request processing failed; nested exception is org.skife.jdbi.v2.exceptions.UnableToCreateStatementException: org.postgresql.util.PSQLException: This connection has been closed. [statement:"sql/sample-id-select", located:" /* Find sample id / SELECT s.id FROM sample_ids s WHERE s.sample_id = :sampleId AND s.project_id = :projectId ", rewritten:" / Find sample id / SELECT s.id FROM sample_ids s WHERE s.sample_id = ? AND s.project_id = ? ", arguments:{ positional:{}, named:{sampleId:'PCSI_0101_Si_R',projectId:'PACA-CA'}, finder:[]}]

type Exception report

message Request processing failed; nested exception is org.skife.jdbi.v2.exceptions.UnableToCreateStatementException: org.postgresql.util.PSQLException: This connection has been closed. [statement:"sql/sample-id-select", located:" / Find sample id / SELECT s.id FROM sample_ids s WHERE s.sample_id = :sampleId AND s.project_id = :projectId ", rewritten:" / Find sample id / SELECT s.id FROM sample_ids s WHERE s.sample_id = ? AND s.project_id = ? ", arguments:{ positional:{}, named:{sampleId:'PCSI_0101_Si_R',projectId:'PACA-CA'}, finder:[]}]

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.skife.jdbi.v2.exceptions.UnableToCreateStatementException: org.postgresql.util.PSQLException: This connection has been closed. [statement:"sql/sample-id-select", located:" / Find sample id / SELECT 	s.id  FROM 	sample_ids s WHERE  	s.sample_id  = :sampleId AND  	s.project_id = :projectId ", rewritten:" / Find sample id / SELECT 	s.id  FROM 	sample_ids s WHERE  	s.sample_id  = ? AND  	s.project_id = ? ", arguments:{ positional:{}, named:{sampleId:'PCSI_0101_Si_R',projectId:'PACA-CA'}, finder:[]}]

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)

org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)

javax.servlet.http.HttpServlet.service(HttpServlet.java:622)

org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

root cause

org.skife.jdbi.v2.exceptions.UnableToCreateStatementException: org.postgresql.util.PSQLException: This connection has been closed. [statement:"sql/sample-id-select", located:" / Find sample id / SELECT 	s.id  FROM 	sample_ids s WHERE  	s.sample_id  = :sampleId AND  	s.project_id = :projectId ", rewritten:" / Find sample id */ SELECT 	s.id  FROM 	sample_ids s WHERE  	s.sample_id  = ? AND  	s.project_id = ? ", arguments:{ positional:{}, named:{sampleId:'PCSI_0101_Si_R',projectId:'PACA-CA'}, finder:[]}]

org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1308)

org.skife.jdbi.v2.Query.fold(Query.java:170)

org.skife.jdbi.v2.Query.first(Query.java:265)

org.skife.jdbi.v2.Query.first(Query.java:257)

org.skife.jdbi.v2.sqlobject.ResultReturnThing$SingleValueResultReturnThing.result(ResultReturnThing.java:111)

org.skife.jdbi.v2.sqlobject.ResultReturnThing.map(ResultReturnThing.java:47)

org.skife.jdbi.v2.sqlobject.QueryHandler.invoke(QueryHandler.java:44)

org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:171)

org.skife.jdbi.v2.sqlobject.SqlObject$1.intercept(SqlObject.java:75)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.getSampleId(<generated>)

org.icgc.dcc.id.server.repository.SampleRepository.getId(SampleRepository.java:36)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.CGLIB$getId$5(<generated>)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de$$FastClassByCGLIB$$9c06dca.invoke(<generated>)

org.skife.jdbi.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)

org.skife.jdbi.v2.sqlobject.PassThroughHandler.invoke(PassThroughHandler.java:36)

org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:171)

org.skife.jdbi.v2.sqlobject.SqlObject$1.intercept(SqlObject.java:75)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.getId(<generated>)

org.icgc.dcc.id.server.repository.BaseRepository.resolveId(BaseRepository.java:62)

org.icgc.dcc.id.server.repository.BaseRepository.findId(BaseRepository.java:50)

org.icgc.dcc.id.server.repository.SampleRepository.findId(SampleRepository.java:28)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.CGLIB$findId$1(<generated>)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de$$FastClassByCGLIB$$9c06dca.invoke(<generated>)

org.skife.jdbi.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)

org.skife.jdbi.v2.sqlobject.PassThroughHandler.invoke(PassThroughHandler.java:36)

org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:171)

org.skife.jdbi.v2.sqlobject.SqlObject$1.intercept(SqlObject.java:75)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.findId(<generated>)

org.icgc.dcc.id.server.controller.SampleController.sampleId(SampleController.java:60)

org.icgc.dcc.id.server.controller.SampleController$$FastClassBySpringCGLIB$$2b3087c8.invoke(<generated>)

org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)

org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:345)

org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:414)

org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:327)

org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)

org.icgc.dcc.id.server.controller.SampleController$$EnhancerBySpringCGLIB$$cde09fab.sampleId(<generated>)

sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)

javax.servlet.http.HttpServlet.service(HttpServlet.java:622)

org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

root cause

org.postgresql.util.PSQLException: This connection has been closed.

org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)

org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)

org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)

sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)

org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)

org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)

com.sun.proxy.$Proxy79.prepareStatement(Unknown Source)

org.skife.jdbi.v2.DefaultStatementBuilder.create(DefaultStatementBuilder.java:47)

org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1304)

org.skife.jdbi.v2.Query.fold(Query.java:170)

org.skife.jdbi.v2.Query.first(Query.java:265)

org.skife.jdbi.v2.Query.first(Query.java:257)

org.skife.jdbi.v2.sqlobject.ResultReturnThing$SingleValueResultReturnThing.result(ResultReturnThing.java:111)

org.skife.jdbi.v2.sqlobject.ResultReturnThing.map(ResultReturnThing.java:47)

org.skife.jdbi.v2.sqlobject.QueryHandler.invoke(QueryHandler.java:44)

org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:171)

org.skife.jdbi.v2.sqlobject.SqlObject$1.intercept(SqlObject.java:75)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.getSampleId(<generated>)

org.icgc.dcc.id.server.repository.SampleRepository.getId(SampleRepository.java:36)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.CGLIB$getId$5(<generated>)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de$$FastClassByCGLIB$$9c06dca.invoke(<generated>)

org.skife.jdbi.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)

org.skife.jdbi.v2.sqlobject.PassThroughHandler.invoke(PassThroughHandler.java:36)

org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:171)

org.skife.jdbi.v2.sqlobject.SqlObject$1.intercept(SqlObject.java:75)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.getId(<generated>)

org.icgc.dcc.id.server.repository.BaseRepository.resolveId(BaseRepository.java:62)

org.icgc.dcc.id.server.repository.BaseRepository.findId(BaseRepository.java:50)

org.icgc.dcc.id.server.repository.SampleRepository.findId(SampleRepository.java:28)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.CGLIB$findId$1(<generated>)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de$$FastClassByCGLIB$$9c06dca.invoke(<generated>)

org.skife.jdbi.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)

org.skife.jdbi.v2.sqlobject.PassThroughHandler.invoke(PassThroughHandler.java:36)

org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:171)

org.skife.jdbi.v2.sqlobject.SqlObject$1.intercept(SqlObject.java:75)

org.icgc.dcc.id.server.repository.SampleRepository$$EnhancerByCGLIB$$fd9c25de.findId(<generated>)

org.icgc.dcc.id.server.controller.SampleController.sampleId(SampleController.java:60)

org.icgc.dcc.id.server.controller.SampleController$$FastClassBySpringCGLIB$$2b3087c8.invoke(<generated>)

org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)

org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:345)

org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:414)

org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:327)

org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)

org.icgc.dcc.id.server.controller.SampleController$$EnhancerBySpringCGLIB$$cde09fab.sampleId(<generated>)

sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)

org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)

javax.servlet.http.HttpServlet.service(HttpServlet.java:622)

org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

note The full stack trace of the root cause is available in the Apache Tomcat/8.5.5 logs.

Apache Tomcat/8.5.5

.
[INFO ] Logging out the session

SOP Appendix, provide more detail on list of Reference Chromosomes

Reference chromosomes can be indicated in the analysis xml. This is not a requirement, one can simply provide the code for the reference genome. I've not included these chromosomes for my current submissions using the egasub tool, but i have included in submissions i had done outside of this tool,

The need to include these is stronger if alignments are against specific chromosomes only,

The appendix lists the Tags and Accessions for each of the reference chromosomes, but using this document it is difficult to know precisely which builds or chromosome these apply to. Aligned sequence data files don't necessarily indicate these accessions.

I'm suggesting that this section of the SOP be expanded to provide a bit more detail so the user does not have to look this up elsewhere. ie. in addition to the accession, also provide the build and chromosome name (or other identifiers)

eg.
CM000663.2 GRCh38 Homo sapiens chromosome 1
(https://www.ncbi.nlm.nih.gov/nuccore/CM000663.2)

Neccessity of running dry_run prior to submission

dry_run is useful to test out the submission, but I typically setup a process to submit many objects at one time. after i've done the dry_run on one or a few, then its longer of value

the current tool won't allow "submit" unless "dry_run" has been run first. It would be better if this was not a requirement prior to running "submit"

Missing dataset description

When running "egasub dataset –submit", the submission fails and reports "Dataset's description is missing". This appears to be a new requirement at EGA's end. Audald mentioned the following requirements for a dataset description:

"Dataset description is expected to be a 3 - 4 sentences definition of the dataset content, including sample number and details, file type and technology/experimentation used"

Example of log entry from egasub

Select the dataset type: 0

Enter dataset alias (unique dataset name) [Cagekid]: Cagekid_release27
Enter dataset title: "Cagekid:Cancer Genomics of the Kidney"
[INFO ] Registering 'dataset' ...
[INFO ] Submit 'dataset' ...
[ERROR] Submitting dataset failed: Error occurred while submitting 'dataset':
Submission failed (note that 'File not found' error, if any, will disappear if you make sure file is indeed uploaded and give it a bit more time (could be a few hours) for EGA systems to synchronize file information):

Dataset's description is missing

Aborted!

Example EGA dataset XML with DESCRIPTION tag:

<?xml version = '1.0' encoding = 'UTF-8'?><DATASETS>
   <DATASET accession="EGAD00001003949" alias="DCC_R27_WGS_Runs.2" broker_name="EGA" center_name="OICR_ICGC" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <IDENTIFIERS>
         <PRIMARY_ID>EGAD00001003949</PRIMARY_ID>
         <SUBMITTER_ID namespace="OICR_ICGC">DCC_R27_WGS_Runs.2</SUBMITTER_ID>
      </IDENTIFIERS>
      <TITLE>ICGC PRAD-CA WGS fastq</TITLE>
      <DESCRIPTION>Raw lane level fastq files from Whole genome sequencing in support of ICGC PRAD-CA Variant calls</DESCRIPTION>

Dataset description

When submitting a dataset, EGA requires a dataset description. I don't recall if this was requested when I ran the egasub tool, but when I asked for the release of the dataset I generated through egasub, EGA helpdesk came back and asked for this information.

Either it's not being requested during dataset formation, or its not being uploaded properly.

running egasub on xfer.res.oicr.on.ca

Hi Junjun,
I'm back to using this tool again, after a long absence.

I consistently get this warning when I run it. (below). It doesn't appear to prevent me from processing the data, but it would be good to fix.

Is this something I need to upgrade to fix, or should I be discussing this with IT?

L

/u/lheisler/.local/venvs/egasub/lib/python2.6/site-packages/requests/packages/urllib3/util/ssl_.py:132: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

submission receipt with EGA accessions

After submission, there is a message indicating success both on the terminal and in the logs. But the accessions that are generated from the submission are not returned.

This is a feature of the REST submission system at EGA.

One can query for the accessions through the api, but there is a delay before this is available. It would be good to report back the accession that was generated so it is available immediately

update github tag "ega" to "ega-archive"

This Repo uses "ega" as a github tag/"topic".
Other projects in the "ega" topic use "ega" are primarily emulators and retro computer things, refering to the "Enhanced Graphics Adapter" (cf. vga/svga)

The primary EGA-archive seems to prefer the "ega-archive" tag (although they also use "EGA")

  • to avoid confusion it would be good to add the "ega-archive" tag on this repo.
  • Removal of the "ega" tag might lessen confusion with retro gaming and operating system tools

'alias' can not include special characters

It's been observed during testing, although . (dot) is allowed to be in the dataset alias (eg, alignment.20170115) when create a new dataset. However, when query the dataset using the same alias, the existing dataset failed return. It turns out the EGA query API endpoint stripped off everything after the dot, eg, alignment.20170115 became alignment.

Solution: add local validation to ensure alias not include special characters. The regex ICGC DCC uses: ^[\w\-_]+$

consistent command line usage for dataset submission

Not a major issue but it would be good to have a consistent command line form for different submission types
in particular

  1. for analyzed or unaligned data
    egasub dry_run
    egasub submit

  2. for the dataset this is dfferent
    egasub dataset --dry_run
    egasub dataset --submit

why not:
egasub dry_run dataset
egasub submit dataset

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.