Spring Cloud Data Flow Implementation for Cloud Foundry

Home Page: http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry/

License: Apache License 2.0

Java 45.61% Ruby 0.55% CSS 7.41% XSLT 43.07% HTML 3.36%

spring-cloud-dataflow-server-cloudfoundry's Introduction

spring-cloud-dataflow-server-cloudfoundry is no longer actively maintained by VMware, Inc.

Spring Cloud Data Flow Server for Cloud Foundry

This project provides support for deploying Spring Cloud Data Flow's streaming and batch data pipelines to Cloud Foundry. It includes an implementation of Spring Cloud Data Flow’s Deployer SPI for Cloud Foundry.

Please refer to the reference documentation on how to get started.

NOTE: Spring Cloud Data Flow v2.0

Beginning with Spring Cloud Data Flow v2.0.0, we have consolidated the Local, Cloud Foundry, and Kubernetes servers into a single server. Instead of building, maintaining, and shipping individual server-implementations, we have consolidated them into a single server, and all the supported platform-specific deployer implementation libraries are bundled in it.

That would mean, as a user, you don’t need to pick different JARs or Docker-images depending on what platform you want to run.

The other notable benefit to this approach is the ability to configure the single-server with multiple platform backends. The same server can be configured against Local, "n" number of Cloud Foundry org/space combinations, and as well against "n" number of Kubernetes clusters and the associated namespace. At the time of stream-deployment or task-launches, you would be able to pick-and-choose the platform where you want to orchestrate the deployment. It would further simplify and promote CI/CD practices for data pipelines.

The getting-started steps for each platform remains mostly the same. One significant change is how Tasks are configured, as we are opening up the capability for Tasks to be launched across different platforms as well.

In summary, please DO NOT use the master branch of this repository; switch over to Spring Cloud Data Flow v2.0 instead. We intend to continue to support the Spring Cloud Data Flow’s v1.x branch of CloudFoundry-server for a period of at least a year from the general availability of v2.0 GA release.

Contributing

We love contributions. Follow this link for more information on how to contribute.

Building

Clone the repo and type

$ ./mvnw clean install

For more information on building, see this link.

spring-cloud-dataflow-server-cloudfoundry's People

Contributors

Stargazers

Watchers

spring-cloud-dataflow-server-cloudfoundry's Issues

Remove CF connector library dependency

As a developer, I'd like to migrate off of the Cloud Foundry connector library; instead, I'd want to take advantage of VcapEnvironmentPostProcessor in Boot that parses the environment variables as json format into a flat list of java key-value pairs.

other items to consider:

A variation/extension of this would translate that flat list of java property key-value pairs into the well known boot properties that are used to create datasource, redis or rabbit connection objects.
At runtime, one would need to specify a hint as to the services that are to be used to aid in the name translation.
Perhaps even an API lookup to see what service names have been bound to the app would be sufficient.

Implement CF TaskLauncher

As a developer, I'd like to implement 'TaskLauncher' for Cloud Foundry server, so the contract is consistent/aligned with spring-cloud-deployer-cloudfoundry contracts.

Acceptance:

I'm able to launch timestamp task successfully
Unit tests included
Integration tests included

Modules should use unique routes

If I deploy a ticktock stream the modules are deployed with non-unique routes like http://ticktock-time.cfapps.io. On PWS there is a strong likelihood that someone else has tried deploying ticktock already. We should use unique/random routes for these modules.

Add support to launch tasks

As a user, I'd want to execute spring-cloud-task on Diego.

Add a DataSource for the task database

The spring-cloud-dataflow-admin-starter now creates the task database tables on start-up. All SPI implementations should add a DataSource to their configuration to be used for the tasks.

Stream name is missing from deployed apps

As a user, I deployed a new stream (using new CF deployer) named foo=time | log and I was hoping to the see the apps foo-time and foo-log; instead I see them with the names time and log respectively. This would instantly break if I deploy another stream that includes similar apps; because, the app names must be unique in CF.

Acceptance:

I'm able to deploy time | log multiple times
Each deployed app includes stream name prefix (eg: foo-time, where foo is the stream name and time is the app)

Review and update CF SPI reference guide

As a developer, I'd like to review and update CF SPI deployer section(s) in the reference guide.

Study Task implementation using CAPI V3 APIs

As a developer, I'd like to explore options to implement Task support in SCDF (on PCF), so I can come up with design proposals for review.

Acceptance:

POC / prototype available for review
Design options documented

Explore options to bind CUPS at module level

As a user, I'd like to bind to user-provided-service that's running outside of PCF, so I can connect to external services such as: hadoop. Typically, any service that's not available in CF marketplace.

The scope of this spike is to study the options to bind CUPS at 'module level' as opposed to wiring it at the data flow admin application. We do not want to expose the bind to every module launched/orchestrated by admin.

Acceptance:
Design proposal for admin including DSL and Shell support is documented

Restructure CF projects

As a developer, I'd like to emulate the restructuring of Local Server and adapt the CF implementation, so they're all aligned with consistent project structure/assembly.

Acceptance:
Following three projects are available

spring-cloud-dataflow-deployer-cf
spring-cloud-starter-dataflow-server-cf
spring-cloud-dataflow-server-cf

note: depending on the naming decision [i.e. spring-cloud-dataflow-deployer-cf vs. spring-cloud-cf], we may have to host the cf deployer in its own repo

Using SpEL expression fails in CF

As a user, I'm trying a simple stream with a transform processor, but the application fails to bootstrap with an exception. This stream works on local SPI.

stream:

stream create foo --definition "http | transform --expression=('Lower Case: ').concat(payload).concat(' Upper Case: ').concat(payload.toUpperCase()) | log" --deploy

error:

=>"70879506-5493-4399-a65e-24ea3857826e"}
2016-02-25T07:45:21.81-0800 [CELL/0]     OUT Creating container
2016-02-25T07:45:22.26-0800 [CELL/0]     OUT Successfully created container
2016-02-25T07:45:27.28-0800 [APP/0]      ERR bash: -c: line 0: unexpected EOF while looking for matching `''
2016-02-25T07:45:27.29-0800 [APP/0]      ERR bash: -c: line 1: syntax error: unexpected end of file
2016-02-25T07:45:27.29-0800 [APP/0]      OUT Exit status 1
2016-02-25T07:45:27.29-0800 [CELL/0]     OUT Exit status 2
2016-02-25T07:45:27.31-0800 [API/0]      OUT App instance exited with guid 992e2ae3-7230-4504-9abc-9073c1e3da7f payload: {"instance"=>"50d9aa81-6e79-4414-5fe6-9f8958de8981", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* 2 error(s) occurred:\n\n* Exited with status 1\n* cancelled\n* cancelled", "crash_count"=>2, "crash_timestamp"=>1456415127310691077, "version"=>"70879506-5493-4399-a65e-24ea3857826e"}
2016-02-25T07:45:27.36-0800 [CELL/0]     OUT Creating container
2016-02-25T07:45:28.06-0800 [CELL/0]     OUT Successfully created container
2016-02-25T07:45:29.30-0800 [APP/0]      ERR bash: -c: line 0: unexpected EOF while looking for matching `''
2016-02-25T07:45:29.32-0800 [APP/0]      ERR bash: -c: line 1: syntax error: unexpected end of file
2016-02-25T07:45:29.33-0800 [APP/0]      OUT Exit status 1
2016-02-25T07:45:29.34-0800 [CELL/0]     OUT Exit status 2
2016-02-25T07:45:29.36-0800 [API/0]      OUT App instance exited with guid 992e2ae3-7230-4504-9abc-9073c1e3da7f payload: {"instance"=>"6cd5a9ec-25ef-42c7-5b7e-a6c5044df6a0", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* 2 error(s) occurred:\n\n* Exited with status 1\n* cancelled\n* cancelled", "crash_count"=>3, "crash_timestamp"=>1456415129350017510, "version"=>"70879506-5493-4399-a65e-24ea3857826e"}
2016-02-25T07:46:20.38-0800 [CELL/0]     OUT Creating container
2016-02-25T07:46:21.15-0800 [CELL/0]     OUT Successfully created container
2016-02-25T07:46:22.41-0800 [APP/0]      ERR bash: -c: line 0: unexpected EOF while looking for matching `''
2016-02-25T07:46:22.41-0800 [APP/0]      ERR bash: -c: line 1: syntax error: unexpected end of file
2016-02-25T07:46:22.42-0800 [APP/0]      OUT Exit status 1
2016-02-25T07:46:22.42-0800 [CELL/0]     OUT Exit status 2
2016-02-25T07:46:22.44-0800 [API/0]      OUT App instance exited with guid 992e2ae3-7230-4504-9abc-9073c1e3da7f payload: {"instance"=>"9834eab0-51f4-4a87-5968-2fd545e140c1", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* 2 error(s) occurred:\n\n* Exited with status 1\n* cancelled\n* cancelled", "crash_count"=>4, "crash_timestamp"=>1456415182432067205, "version"=>"70879506-5493-4399-a65e-24ea3857826e"}

This could be related to regex issue observed in spring-cloud/spring-cloud-stream-modules#195

Implement CF App deployer

As a developer, I'd like to refactor CF deployer to adapt to cloudfoundry-java-client V2 APIs, so the newer AppDeployer APIs improve the overall CF deployment experience.

Acceptance:

I'm able to successfully deploy time | log stream
I'm able to tail the logs for each microservice application successfully
I can query the status of each application successfully
When the stream is destroyed, the applications are deleted
I'm able to verify the application status by logging into AppsManager

Document how to use cloud config to configure modules

As a developer, I'd want to document how to use cloud-config to configure modules, so it'd be useful for end users who create streaming pipeline with modules that include login credentials listed as inline attributes. [ex: cassandra, jdbc, ..]

Acceptance:
Documentation include how to configure cloud-config with module specific metadata options such as username and password

Prefix dataflow server unique ID and stream name to the name of deployed applications

I'm trying SCDF Admin CloudFoundry 1.0.0.M1

Stream name seems to have to be unique in the same domain.

Otherwise 400 error will return as follows:

dataflow:>stream create --name ticktock --definition "time | log" --deploy
Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: 400 Bad Request

400 Bad Request

org.springframework.cloud.dataflow.rest.client.DataFlowClientException: 400 Bad Request

    at org.springframework.cloud.dataflow.rest.client.VndErrorResponseErrorHandler.handleError(VndErrorResponseErrorHandler.java:52)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
    at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:357)
    at org.springframework.cloud.dataflow.rest.client.StreamTemplate.createStream(StreamTemplate.java:79)
    at org.springframework.cloud.dataflow.shell.command.StreamCommands.createStream(StreamCommands.java:99)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
    at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:64)
    at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:57)
    at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:130)
    at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
    at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
    at java.lang.Thread.run(Thread.java:745)

Is it correct? Actually, it was success when ticktock-foo was used as the stream name, which may not be taken.

I think it's so inconvenient. Can we use --random-route to escape duplication of the name?

My memo is here.

Getting repeated 503 errors from admin

Whenever I deploy a stream and later issue a "stream list" command I keep getting 503 errors until first stream module is running

dataflow:>stream create ticktock --definition "time | log" --deploy 
Created and deployed new stream 'ticktock'
dataflow:>stream list
Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: 503 Service Unavailable

dataflow:>stream list
Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: 503 Service Unavailable

SpringCloud data flow app doesn't start, getting SSL handshare error

Pushed admin app as per the below link. but getting ssl handshakes error during the start. we did try with default java build pack.

http://docs.spring.io/spring-cloud-dataflow/docs/1.0.0.M1/reference/html/getting-started-deploying-spring-cloud-dataflow.html

Error:
2016-02-26T16:27:42.07-0700 [App/0] ERR Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with nam
e 'taskController' defined in file [/home/vcap/app/org/springframework/cloud/dataflow/admin/controller/TaskController.class]: Unsatisfied dependency e
xpressed through constructor argument with index 2 of type [org.springframework.cloud.dataflow.module.deployer.ModuleDeployer]: : Error creating bean
with name 'taskModuleDeployer' defined in class path resource [org/springframework/cloud/dataflow/module/deployer/cloudfoundry/CloudFoundryModuleDeplo
yerConfiguration.class]: Unsatisfied dependency expressed through constructor argument with index 1 of type [org.springframework.cloud.dataflow.module
.deployer.cloudfoundry.CloudFoundryApplicationOperations]: : Error creating bean with name 'cloudFoundryApplicationOperations' defined in class path r
esource [org/springframework/cloud/dataflow/module/deployer/cloudfoundry/CloudFoundryModuleDeployerConfiguration.class]: Bean instantiation via factor
y method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.dataflow.m
odule.deployer.cloudfoundry.CloudFoundryApplicationOperations]: Factory method 'cloudFoundryApplicationOperations' threw exception; nested exception i
s error="access_denied", error_description="Error requesting access token."; nested exception is org.springframework.beans.factory.BeanCreationExcepti
on: Error creating bean with name 'cloudFoundryApplicationOperations' defined in class path resource [org/springframework/cloud/dataflow/module/deploy
er/cloudfoundry/CloudFoundryModuleDeployerConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.
beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.dataflow.module.deployer.cloudfoundry.CloudFoundryApplicationOperat
ions]: Factory method 'cloudFoundryApplicationOperations' threw exception; nested exception is error="access_denied", error_description="Error request
ing access token."; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'taskModuleDep
loyer' defined in class path resource [org/springframework/cloud/dataflow/module/deployer/cloudfoundry/CloudFoundryModuleDeployerConfiguration.class]:
Unsatisfied dependency expressed through constructor argument with index 1 of type [org.springframework.cloud.dataflow.module.deployer.cloudfoundry.C
loudFoundryApplicationOperations]: : Error creating bean with name 'cloudFoundryApplicationOperations' defined in class path resource [org/springframe
work/cloud/dataflow/module/deployer/cloudfoundry/CloudFoundryModuleDeployerConfiguration.class]: Bean instantiation via factory method failed; nested
exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.dataflow.module.deployer.cloudfoun
dry.CloudFoundryApplicationOperations]: Factory method 'cloudFoundryApplicationOperations' threw exception; nested exception is error="access_denied",
error_description="Error requesting access token."; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'cloudFoundryApplicationOperations' defined in class path resource [org/springframework/cloud/dataflow/module/deployer/cloudfoundry/CloudFou
ndryModuleDeployerConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationE
xception: Failed to instantiate [org.springframework.cloud.dataflow.module.deployer.cloudfoundry.CloudFoundryApplicationOperations]: Factory method 'c
loudFoundryApplicationOperations' threw exception; nested exception is error="access_denied", error_description="Error requesting access token."
2016-02-26T16:27:42.07-0700 [App/0] ERR at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolv

Docs need updating for CF deployment

There are some issues with the docs for deploying to CF:

cf push s-c-dataflow-server --no-start -p spring-cloud-dataflow-server-cloudfoundry-1.0.0.BUILD-SNAPSHOT.jar

should add --random-route or -n <my-cf-server> so there aren't collisions with multiple users deploying Dataflow server to PWS.

The shell command to target the server need to change to use dataflow instead of admin and also include the custom/random route from above:

server-unknown:>dataflow config server http://<my-cf-server>.cfapps.io

Add support to deploy large applications

As a user, I'm deploying a stream composed of both OOTB and customer applications; however, given the size of the application and the default timeout limit of 60s, I'm unable to download and push the application to CF successfully.

Acceptance:

Default settings supported by CF are overridable
I'm able to override CF_STARTUP_TIMEOUT to a value >60s

Custom port shoud NOT be set for modules

When deploying a stream module with a custom --port parameter the CF deployment needs to include this port

Study Spinnaker and integration requirements

As a developer, I'd like to investigate the integration relationship between Spinnaker and SCDF.

Acceptance:

POC and design review
Install Spinnaker on Cloud foundry and deploy v1 of an app followed by v2.

Need to be able to configure health check timeout

The health check timeout is 60s by default. We should be able to override this so we can deploy over slower networks.

Remove ModuleLauncher from CF AppDeployer

As a developer, I'd like to remove ModuleLauncher dependency from CF App deployer.

Routes are not deleted when the stream is destroyed

As a user, I tried destroying the stream time | log, where time and log are Boot applications; however, I still see the routes even after successful delete of the applications.

stream:

stream create foo --definition "time | log" --deploy
stream destroy foo

cf cli:

cf routes
Getting routes as sabby ...

space         host                                  domain                   apps
development   foo-log                               app.navy.springapps.io
development   foo-time                              app.navy.springapps.io

Investigate rolling upgrades using Spinnaker

As a developer, I'd like to do rolling upgrade on an existing application in CF using Spinnaker.

Acceptance:

Create a new version of the log sink application, log-rabbit-1.1.0.jar and upload to the maven repository that the dataflow server points to by default.
Use the spinnaker GUI to replace log-rabbit v1 with log-rabbit v2.

Add null-check when accessing InstanceStats.Usage#getTime()

We should also possibly update the CF Client Java library once a new release is out.

Move CF deployer to spring-cloud-cloudfoundry-deployer

As a developer, I'd want to move implementation of ApplicationModuleDeployer SPI to spring-cloud-cloudfoundry-deployer project, so it can be consumed as library.

Acceptance:

Existing stream and task unit tests run successfully
The project successfully builds
Docs updated

Use AdminProperties in CloudFoundryModuleDeployerConfiguration

Admin properties are not being enabled.

Revert changes for NPE with Date Parsing when new CF client lib is released

Need to revert #20 once it is handled in the underlying librayr.

Spike: Define a way to test modules in development

As a module developer I'd like to be able to test my modules currently in development using the CF admin.

It seems that the current module resolution with BUILD-SNAPSHOT version makes this difficult. It requires a temporary repo defined and added to the CD admin. It also seems to require modification of the module definitions with a specific snapshot timestamp version, or faking the module version to something other than BUILD-SNAPSHOT.

Add standalone admin artifact and CI build for YARN

As a developer, I'd like to create separate CI pipeline for YARN SPI, so I can run incremental builds and validate SCDF features.

Acceptance:

Bamboo infrastructure for YARN SPI is created
Unit / Integration tests are included in CI plan

Add JUnit Rule for conditional execution of tests

Autoconfigure CF Admin Deployer

As a developer, I'd like to Autoconfigure CF Admin deployer, so I don't have to do component scanning to discover and register beans at each SPI level.

CF SPI: Add support to launch external JVM for tasks

As a user, I'd want to launch an external JVM for tasks when using CF SPI.

Review Spinnaker's CloudDriver specs

As a developer, I'd like to review Spring's 'application deployer' and 'artifact downloader' SPIs for use inside Spinnaker’s cloud driver for Cloud Foundry.

Acceptance:
Determine gaps/overlaps in Spring’s Application Deployer

Add reference documentation

404 on M1 release download

As a user, I'm trying to prep SCDF Admin for CF deployment, but I get a 404 on downloading the M1 release from the reference guide.

Add CI plan for CF deployer

As a developer, I'd like to setup CI plan for CF deployer, so we can validate test suites on incremental code commits.

Change references of `admin` to `server`

This repo has several references of admin and they need to change to use server.

The scope of this issue is limited only to change name of the projects and other references inside the project. and, we need to decide on the name of the entire repo name (probably creating a new repo).

Improve error message propagation

As a user, I've been trying to create a stream on PEZ/PWS, but I always ended up with arbitrary 400 error message in the shell. While looking through the admin logs, I could see something about "not enough resource available". It'd be nice to propagate more details about the deployment errors.

note: this might have to wait till java-client v2 release

Fix parent pom relationships

Current project layout is not self sufficient, as some parent poms are mis-located (and would need to be downloaded from a remote repository, while they're locally available at ../pom.xml)

Can't deploy stream (502 at service binding time)

I keep getting the following error while trying to deploy a stream:

org.springframework.web.client.HttpServerErrorException: 502 Bad Gateway
    at org.cloudfoundry.client.lib.rest.CloudControllerResponseErrorHandler.handleError(CloudControllerResponseErrorHandler.java:26) ~[cloudfoundry-client-lib-1.1.3.jar:na]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.cloudfoundry.client.lib.rest.LoggingRestTemplate.doExecute(LoggingRestTemplate.java:54) ~[cloudfoundry-client-lib-1.1.3.jar:na]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:357) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.cloudfoundry.client.lib.rest.CloudControllerClientImpl.doBindService(CloudControllerClientImpl.java:1742) ~[cloudfoundry-client-lib-1.1.3.jar:na]
    at org.cloudfoundry.client.lib.rest.CloudControllerClientImpl.updateApplicationServices(CloudControllerClientImpl.java:1603) ~[cloudfoundry-client-lib-1.1.3.jar:na]
    at org.cloudfoundry.client.lib.rest.CloudControllerClientImpl.createApplication(CloudControllerClientImpl.java:1183) ~[cloudfoundry-client-lib-1.1.3.jar:na]
    at org.cloudfoundry.client.lib.CloudFoundryClient.createApplication(CloudFoundryClient.java:252) ~[cloudfoundry-client-lib-1.1.3.jar:na]
    at org.springframework.cloud.dataflow.module.deployer.cloudfoundry.ApplicationModuleDeployer$2.run(ApplicationModuleDeployer.java:105) ~[classes/:na]
    at org.springframework.cloud.dataflow.module.deployer.cloudfoundry.Undoer$1.call(Undoer.java:41) ~[classes/:na]
    at org.springframework.cloud.dataflow.module.deployer.cloudfoundry.Undoer$1.call(Undoer.java:38) ~[classes/:na]
    at org.springframework.cloud.dataflow.module.deployer.cloudfoundry.Undoer$UndoerRunner.andUndoBy(Undoer.java:56) ~[classes/:na]
    at org.springframework.cloud.dataflow.module.deployer.cloudfoundry.ApplicationModuleDeployer.deploy(ApplicationModuleDeployer.java:107) ~[classes/:na]
    at org.springframework.cloud.dataflow.admin.controller.StreamDeploymentController.deployStream(StreamDeploymentController.java:185) ~[classes/:na]
    at org.springframework.cloud.dataflow.admin.controller.StreamDeploymentController.deploy(StreamDeploymentController.java:144) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) ~[spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) ~[spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:242) [spring-boot-actuator-1.3.1.BUILD-20151215.152328-92.jar:1.3.1.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111) [spring-boot-actuator-1.3.1.BUILD-20151215.152328-92.jar:1.3.1.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:207) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.1.BUILD-20151215.152328-92.jar:1.3.1.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.30.jar:8.0.30]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

Add CI pipeline for SCDF-Admin-CF

As a developer, I'd like to create separate CI pipeline for CF SPI, so I can run incremental builds and validate SCDF features.

Acceptance:

Bamboo infrastructure for CF SPI is created
Unit / Integration tests are included in CI plan

() seems to need to be escaped in DSL

Stream http | transform --expression=payload.toUpperCase() | log does not work in CF environment.

From error log, payload.toUpperCase() seems to need to be escaped.

2016-02-06T01:34:59.80+0900 [APP/0]      ERR bash: eval: line 0: syntax error near unexpected token `('
2016-02-06T01:34:59.80+0900 [APP/0]      ERR bash: eval: line 0: `exec /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -XX:MetaspaceSize=104857K -Xss1M -Xmx768M -XX:MaxMetaspaceSize=104857K -Xms768M -cp /home/vcap/app/.:/home/vcap/app/.java-buildpack/spring_auto_reconfiguration/spring_auto_reconfiguration-1.10.0_RELEASE.jar org.springframework.boot.loader.JarLauncher --args.0.spring.cloud.stream.bindings.input.destination=demodemo.0 --args.0.spring.cloud.stream.bindings.output.destination=demodemo.1 --args.0.spring.jmx.default-domain=demodemo.transform --modules=org.springframework.cloud.stream.module:transform-processor:jar:exec:1.0.0.M2 --args.0.expression=payload.toUpperCase()'

http | transform --expression=payload.toUpperCase\(\) | log worked fine.
This means DSL depends on the platform. I think it's confusing,

Allow per-environment timeout customizations

This is to give a chance to some tests on the CI environment to finish

Rename admin-starter dependency

As a developer, I'd like to pull spring-cloud-dataflow-server-core instead of spring-cloud-dataflow-admin-starter as dependency.

Relates to spring-cloud/spring-cloud-dataflow#349.

Module deployments fail on CF

Creating a ticktock stream the module deployments fail with:

2016-03-03T16:55:26.000+00:00 [APP] ERR Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is org.springframework.context.ApplicationContextException: Failed to start bean 'inputBindingLifecycle'; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

Destroying a stream does not delete the associated apps

As a user, I tried destroying a stream using the 'new' CF deployer. Though the operation is successful, I'm still able to list the apps in CF CLI and in AppsMgr.

Acceptance:
Destroying a stream should delete the associated apps.

Investigate naming conventions to deploy apps with Spinnaker

As a developer, I'd like to investigate naming conventions acceptable by Spinnaker, so I can deploy a ticktock stream on CF using Spinnaker + SCDF.

Acceptance:

Ensure that application naming conventions make this recognized by spinnaker.
This would use log-rabbit-1.0.0.jar

Add integration tests

CF module deployment should wrap arguments with quotes

The module deployment for CF now has issues when arguments contain spaces. We should try wrapping the argument inside quotes.

There is a recent change with the new java buildpack v 3.4 that is a fix to address a process termination issue for Diego - https://www.pivotaltracker.com/n/projects/1003146/stories/103541132

The new buildpack is now adding exec for the java launch, so it seems the argument handling is slightly different from previous buildpack versions. In particular the following now fails:

transform --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT,com.datastax.cassandra:cassandra-driver-core:2.1.5' --expression='new demo.domain.Book(T(com.datastax.driver.core.utils.UUIDs).timeBased(), payload.title, payload.author)'

while this works:

transform --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT,com.datastax.cassandra:cassandra-driver-core:2.1.5' --expression='\"new demo.domain.Book(T(com.datastax.driver.core.utils.UUIDs).timeBased(), payload.title, payload.author)\"'

Notice the quotes used for the --expression argument.

spring-attic / spring-cloud-dataflow-server-cloudfoundry Goto Github PK

spring-cloud-dataflow-server-cloudfoundry's Introduction

spring-cloud-dataflow-server-cloudfoundry is no longer actively maintained by VMware, Inc.

Spring Cloud Data Flow Server for Cloud Foundry

NOTE: Spring Cloud Data Flow v2.0

Contributing

Building

spring-cloud-dataflow-server-cloudfoundry's People

Contributors

Stargazers

Watchers

Forkers

spring-cloud-dataflow-server-cloudfoundry's Issues

Recommend Projects

Recommend Topics

Recommend Org