Depends on update to jetty-9.4 #10 .

from jetty-runtime.

Discussed in a hangout and reproduced here for the record, with some extra detail:

• standard jetty-9.4 session clustering includes options for caching sessions on a node. Benefit to this is that simultaneous requests for the same context are able to share the same session object. Works best with a sticky load balancer. Without a sticky loadbalancer there are configuration options to alleviate problem of session freshness:
  • evict the session after all concurrent requests have completed
  • evict idle sessions
    None of the above prevent the scenario of multiple simultaneous requests being directed to 2 different instances (without a sticky loadbalancer) and thus 2 different copies of the session. However, without such a solution apps that do experience multiple simultaneous requests may not work (see GoogleCloudPlatform/appengine-java-vm-runtime#151).

Decision was not to use this caching ability at least at first due to lack of sticky loadbalancer. May investigate upgrading to use a session cache (with appropriate eviction strategy) later.

from jetty-runtime.

It's worth enumerating the different session setups that are in play and proposed so that we can understand a) what we need to do for jetty-9.4 to support existing (compat) session management and b) what we need to do for new jetty-runtime 9.4 session management options.

For jetty-runtime we want to introduce full-featured session management, whilst for compat-runtime we need to be 100% backwards compatible with session data format, and as backwards-compatible as possible in terms of functionality: in other words the initial aim is to be "bug-for-bug" compatible with the current compat-runtime session management. Future iterations may introduce more advanced session features in order to provide enhanced behaviour/fixes for existing bugs in compat session management.

I'm currently working on porting the legacy gcloud datastore sessions (described below) to jetty-9.4, and will next work on porting the no-op sessions.

no-op sessions

Used in compat when the user does not explicitly enable sessions in appengine-web.xml.
Characteristics:

• bare minimum session functionality for jsps to work
• no session-sharing for simultaneous requests
• no caching
• no persistence
• no expiry
• no scavenging

legacy gcloud datastore sessions

Used in compat when sessions-enabled=true in appengine-web.xml.
Characteristics:

• single context only
• no session-sharing for simultaneous requests
• sessions persisted as soon as created
• gcloud datastore persistence fronted by memcached for performance/help with non-sticky load-balancing
• session scavenging by periodic hit to SessionServlet which deletes expired sessions without calling HttpSessionListeners
• changes to session.maxIdleTime not persisted
• need to maintain backwards compatibility with existing persisted session data format

non-clustered sessions

New feature for jetty-runtime.
Characteristics:

• enabled by default
• uses a standard jetty session module
• sessions not clustered/persisted
• sessions cached on local node only
• characteristics of session caching very configurable eg when/if to evict sessions
• only works with single node
• session scavenging by thread in jetty runtime

clustered gcloud datastore sessions

New feature for jetty-runtime.
Characteristics:

• uses a standard jetty session module
• no compatibility with compat session data required
• no session-sharing/caching
• session scavenging by thread in jetty runtime

Issues:

• how to enable gcloud session module from environment variable
• how to allow user to configure session setup such as datastore backoff and retries and frequency of scavenging, see #50.

clustered gcloud datastore sessions with memcached

New feature for jetty-runtime.
Characteristics:

• as above, with memcached added as L2 cache for session data (also as standard jetty module)
  Issues: same as above

from jetty-runtime.

@janbartel Can you suggest some different ways you think it would make sense to select the session management setup via environment variables?

One way, I'm thinking is to just have a generic environment variable to enable optional jetty modules, like ENABLE_JETTY_RUNTIME_MODULES=datastore-sessions|memcache-datastore-sessions|... Another option is to have individual boolean evnironment variables per module like DATASTORE_SESSIONS_ENABLE=true and MEMCACHE_DATASTORE_SESSIONS_ENABLE=true. Are you thinking along similar lines?

from jetty-runtime.

@meltsufin I suppose it may be possible to use environment variable/s to select between a restricted choice of non-clustered vs datastore vs datastore+memcached session management. Caveat though that the google apis have a (transitive?) dependency on commons logging and thus the user would also need to somehow configure how to plumb commons logging into their log setup. Currently with the jetty-distro if a user enables the session-store-gcloud module we flag a dependency on jcl and ask them to also configure an appropriate jcl module (many choices here) to link to their logging.

However, I don't see how we can use environment variables to access all of the configuration that users may legitimately want to tweak on session management. Here's a few examples:

#jetty.sessionScavengeInterval.seconds
#jetty.session.evictionPolicy
#jetty.session.saveOnInactiveEvict
#jetty.session.saveOnCreate
#jetty.session.removeUnloadableSessions
#jetty.session.gcloud.maxRetries
#jetty.session.gcloud.backoffMs
#jetty.session.gcloud.model.kind
#jetty.session.gcloud.model.id
#jetty.session.gcloud.model.contextPath
#jetty.session.gcloud.model.vhost
#jetty.session.gcloud.model.accessed
#jetty.session.gcloud.model.lastAccessed
#jetty.session.gcloud.model.createTime
#jetty.session.gcloud.model.cookieSetTime
#jetty.session.gcloud.model.lastNode
#jetty.session.gcloud.model.expiry
#jetty.session.gcloud.model.maxInactive
#jetty.session.gcloud.model.attributes
#jetty.session.memcached.expirySec

In addition, I don't see environment variables as being a viable generic mechanism for enabling other jetty modules and tweaking their configurations.

from jetty-runtime.

@janbartel I think the idea is to just provide good defaults so that the majority of users wouldn't need to tweak these settings. Those users who need to set these settings would have to write a custom Dockerfile extending our runtime and providing the configuration file for the module(s).
Does that make sense?

from jetty-runtime.

@meltsufin progress so far: I've been testing with an environment variable called "SESSIONS". The valid values are:

• "datastore" (selects datastore-backed clustered sessions)
• "memcached-datastore" (selects datastore-backed clustered sessions with memcached cache for efficiency)
• if not set, or set to some other value, the default jetty non-clustered sessions are selected.

For now, I've plumbed up the jcl->slf4j-simple logging for the server.

As you know, I've been testing in the "vm:true" environment with a custom runtime. Aside from issue 149 (after a lapse of 24hrs it seems not to be reproduceable!), tests are going well. Sessions are visible in both memcached and gcloud datastore, and being expired appropriately by our scavenger thread.

I will also test "env:flex", once I get white-listing.

Note that the jetty gcloud datastore clustered sessions allows for multitenancy: we allow the jetty property "jetty.session.gcloud.namespace" to select a datastore namespace for the sessions. I could allow this to be set via another environment variable, so it would be amenable to being set in app.yaml. WDYT? Any suggestions for a name? I'd like to bake it into the jetty-9.4 release, so let me know your thoughts on this.

Also, would you like me to create a branch for the jetty-9.4 integration, rather than do this only by pull request? That way you have a bit more visibility of what's going on. I could make the branch once we have jetty-9.4.rc1 released so it does not depend on a snapshot.

from jetty-runtime.

@janbartel Sounds like you're making a lot of progress with this!

The environment variable sounds fine, at least for now.

What's the goal of the jcl->slf4j-simple plumbing for the server?

Have someone requested the env:flex white-listing for you? Let me know if you need us to push it along.

What are you using currently as the default sessions namespace? Does it make sense to make it scoped to the application instance, which would be identified by {service}+{version} in App Engine, or just make it completely global to the project? If we were to allow configuring it, maybe SESSIONS_NAMESPACE is a good environment variable name for it?

Yes, please create a branch for jetty-9.4 integration in this repo. Eventually, I think we'll just add it to master under /jetty9.4/.

from jetty-runtime.

@meltsufin yes please, could you please chase up the white-listing?

The jcl->slf4j-simple plumbing is because gcloud datastore has dependencies on jcl, and that logging in jetty handled by using slf4j. When using the jetty distro, we ask the user to enable their slf4j impl of choice, but as that's not possible in this situation, I've arbitrarily used slf4j-simple. Note that gcloud datastore has released v 0.4.0 with less dependencies, which may or may not have removed the jcl dependency: I will be testing that this week.

As for the namespace, currently it is blank, meaning all deployed apps from a project share the default namespace. Thinking on this a little more, I don't think it's worthwhile introducing another environment variable to set this up for 2 reasons: 1. we don't want an explosion of environment variables and 2. if you're using multitenancy it's highly probable there other things you want to configure, so you'll be needing a custom Dockerfile anyway.

from jetty-runtime.

The jcl->slf4j-simple plumbing is because gcloud datastore has dependencies on jcl, and that logging in jetty handled by using slf4j. When using the jetty distro, we ask the user to enable their slf4j impl of choice, but as that's not possible in this situation, I've arbitrarily used slf4j-simple. Note that gcloud datastore has released v 0.4.0 with less dependencies, which may or may not have removed the jcl dependency: I will be testing that this week.

Assuming the JCL dependency has not been removed, we should perhaps consider always enabling the jetty-slf4j module - regardless of the session settings. This would give us deterministic behaviour as the jetty logging would always go via slf4j regardless of if gcloud sessions are used or not. Otherwise it may be a bit of a surprise if the session are enabled and the logging setup changes.

Once we go via slf4j, that doesn't actually set any log handlers, so we can choose any that we like: slf4j-simple just to put it out stderr or Java Util Logging (configured to go out stderr) if you want a standard? - let's discuss in the design document :)

from jetty-runtime.

@meltsufin FYI in gcloud datastore 0.4.0 the commons logging dependency still exists:

[INFO] +- com.google.cloud:google-cloud-datastore:jar:0.4.0:compile
[INFO] | +- com.google.cloud:google-cloud-core:jar:0.4.0:compile
[INFO] | | +- com.google.http-client:google-http-client:jar:1.21.0:compile
[INFO] | | | - org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] | | | +- commons-logging:commons-logging:jar:1.1.1:compile

from jetty-runtime.

@meltsufin thanks for the whitelisting.

I've been able to start testing "env:flex" with gcloud datastore backed sessions. So far, so good! Sessions are created, persisted and scavenged appropriately.

from jetty-runtime.

@janbartel That's great news! Can you link up to the area where you're developing the code? Thanks!

from jetty-runtime.

FYI

I'm linking this issue to all of the gcloud datastore session work done over in jetty so you have visibility:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=479832
jetty/jetty.project#658
jetty/jetty.project#663
jetty/jetty.project#914
jetty/jetty.project#931
jetty/jetty.project#953
jetty/jetty.project#954
jetty/jetty.project#1007

from jetty-runtime.

@meltsufin I've pushed branch issue-41 so you can see progress. This branch refers to jetty-9.4.0.RC1. Tested it on env:flex and so far, so good.

NB 1. I'm currently using jcl->jul logging instead of slf4j-simple. Easily changeable, just need to firm up the choice of one or another.

NB 2. We will be doing a jetty-9.4.0RC2 next week, after which I'll update the branch.

from jetty-runtime.

Branch updated to jetty-9.4.2-SNAPSHOT (awaiting jetty-9.4.2 release this week), and also updated to gcloud-datastore 0.9.2-beta.

from jetty-runtime.

Now updated to 0.10.0-beta and jetty-9.4.3.

from jetty-runtime.

To recapitulate what has been done:

Session management is enabled by the system property "SESSIONS". The supported values are "datastore", "memcached-datastore" or unset.

If unset, sessions are kept in-memory only and never persisted. Expired sessions are scavenged at a default interval of 10mins.

If set to "datastore", sessions are persisted to gcloud datastore and thus visible to other nodes, but never cached in memory in any node. Sessions are scavenged at the default interval of every 10 mins.

The "memcached-datastore" value is for the future when memcached is available in the flex environment. This setting will use memcached as a distributed cache for session data, with gcloud datastore as the backing store. Sessions are never locally cached on any node. Sessions are scavenged at the default interval of 10mins.

from jetty-runtime.

Status update:
The intention is to remove the need for the special SESSIONS environment variable, and allow the user to use the general purpose JETTY_MODULES_ENABLE variable instead to also configure sessions.

To enable gcloud datastore backed sessions, the app.yaml will look like:
JETTY_MODULES_ENABLE: 'gcp-datastore-sessions'

When memcache support becomes available, the app.yaml to enable it would be:
JETTY_MODULES_ENABLE: 'gcp-memcache-datastore-sessions'

This update depends on upcoming release 9.4.5 of jetty to resolve jetty issue 1494.

from jetty-runtime.

Hi @janbartel I wanted to know whether JETTY_MODULES_ENABLE: 'gcp-memcache-datastore-sessions' is possible on Jetty version "9.4.11.v20180605 9.4-2018-12-17_20_12" for App Engine Flex. Also is there any work in progress to enable Google Cloud MemoryStore for Jetty Session Management in App Engine Flex? Thanks in advance!

from jetty-runtime.