otto-de / edison-microservice Goto Github PK

JavaDocs are missing.
README should be extended
No stand-alone example for edison-cache

Currently we trigger the job execution with the cron scheduling mechanism of spring.
But then all servers in the cluster start the same job at the same time.

It would be better if edison-jobs checks before starting the job, if the job is already running in the cluster. That could be done by asking the database for any running job of the same type.

We use the mongo-jobs plugin, too. Here the best solution would be to create a unique index which assures, that only one running job of a type can exists. If the creating of the job fails, you know, the job has already started on a different server in the cluster.

In the code I found the following todo, which might be connected to this issue:
JobService:101: // TODO: use some kind of database lock so we can prevent race conditions

protected Document byId(final K key) {
    return key != null ? new Document(ID, key.toString()) : new Document();
}

For example if I call AbstractMongoRepository.delete(null) it results in deleting the first document that would be found by matchAll, and it is not possible to delete a document with an _id=null (except it is the first document found by matchAll).

So the implementation should be changed to:

protected Document byId(final K key) {
    return key != null ? new Document(ID, key.toString()) : new Document(ID, null);
}

If edison-mongo is added to the project, a MongoTogglzRepository is instantiated to persist Toggz´ feature state.

Sometimes it might be necessary to use edison-mongo but persist toggles in a different way (or not at all).

In order to support this, the MongoTogglzRepository should be @conditional, and/or an '.*.enabled' property should be supported.

Currently, there is no such requirement, so it will probably not be solved soon. Please notify me, if you need this feature.

Hello, with upgrade to RC4 we get the following exception during the start of a job:
com.mongodb.MongoCommandException: Command failed with error 11000: 'E11000 duplicate key error collection: composing.jobmeta index: _id_ dup key: { : "GenerateNewestCompositionImageJob" }' on server db-live-460925.lhotse.ov.otto.de:27017. The full response is { "ok" : 0.0, "errmsg" : "E11000 duplicate key error collection: composing.jobmeta index: _id_ dup key: { : \"GenerateNewestCompositionImageJob\" }", "code" : 11000 } at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115) at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114) at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:173) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:215) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:152) at com.mongodb.operation.FindAndUpdateOperation$1.call(FindAndUpdateOperation.java:305) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221) at com.mongodb.operation.FindAndUpdateOperation.execute(FindAndUpdateOperation.java:302) at com.mongodb.Mongo.execute(Mongo.java:781) at com.mongodb.Mongo$2.execute(Mongo.java:764) at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:407) at de.otto.edison.mongo.jobs.MongoJobMetaRepository.createValue(MongoJobMetaRepository.java:141) at de.otto.edison.mongo.jobs.MongoJobMetaRepository.setRunningJob(MongoJobMetaRepository.java:70) at de.otto.edison.jobs.service.JobMetaService.aquireRunLock(JobMetaService.java:55) at de.otto.edison.jobs.service.JobService.startAsyncJob(JobService.java:101) at de.otto.social.composing.common.job.JobWatcher.lambda$registerCronJob$3(JobWatcher.java:41) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Unexpected error occurred in scheduled task.

Ob das Problem nur auftritt, wenn bereits alte Jobs von vor dem Release in der DB existieren, weiß ich nicht. Hab jetzt erstmal downgegradet.

In several places, hard-coded /internal URLs are used. Instead of this, the management.context-path property should be respected. Other or no pathes for internal pages should work as well as /internal

Extracting Locking of Jobs from JobRepository
Adding possibility to store meta data, for example to store the read-position of the current import job.

Since upgrading to 1.0.0.RC4 from 1.0.0.RC3 we see the following message in the exception.log:

Inconsistent constructor declaration on bean with name 'jobMutexGroups': single autowire-marked constructor flagged as optional - this constructor is effectively required since there is no default constructor to fall back to: public de.otto.edison.jobs.service.JobMutexGroups(java.util.Set)

The application seems to behave normal. So it is only a confusing message in the log.

The JobRepository interface must have a clearAll() method.

The thymeleaf template status.html renders the vcs url only if it starts with http.
<dd><a th:if="${#strings.startsWith(vcs.url, 'http')}" th:href="${vcs.url}">Open VCS UI</a></dd>

As a consequence, vcs urls like git@git... are not rendered.

RC1 contains a bug that is preventing the configuration of JobStatusCalculators in application.properties

Fewer modules with better support for auto-configuration depending on properties and class path.

...instead of just showing the job id in the uri field

JavaDocs are missing.
README should be extended
No stand-alone example for edison-mongo

Disabling jobs in Edison is easily possible using the lock icon. However, you don't see any reason why a job has been disabled. Working in a team, this means you have to find out who disabled that job and figure out the reason why it was disabled. A simple way to make this more pleasant would be to provide a comment that states why a job has been disabled. The UI could ask for that comment as soon as someone decides to disable a job and could be shown alongside the lock icon for others to see.

See:
https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-1.5-Release-Notes#loggers-endpoint

1. Add a NavBarItem to the Admin NavBar
2. Add a UI to view/configure logging

1. Show last N jobs of a type
2. Show last N jobs with some state
3. Sort by start date / end date
   ...

Job links of status details currently are broken and point to /internal/jobs/ instead of http://...//internal/jobs/

Currently, only the last job for a type is contained in the job overview. While there is a request parameter /internal/jobs?distinct=false, there is
no possibility in the UI to show the last N jobs without adding the parameter manually to the URL.
Because there are new Strategies to calculate job status that are taking more than one job execution into account when calculating the status details, this should changed in that the last 10 jobs for every type is displayed.

The Configuration Bean MongoJobsConfiguration is only active in presence of the Java Class "de.otto.edison.mongo.jobs.configuration.JobConfiguration", which is unavailable.

`
@configuration

@ConditionalOnClass(name = "de.otto.edison.mongo.jobs.configuration.JobConfiguration")

public class MongoJobsConfiguration {...
}`

Use typesafe properties instead of @value annotations in auto configuration of bean.

Especially for failed jobs there should be a button in the UI of a single job, to trigger job execution.

Add the Jobstate skipped for jobs to mark jobs without any business processing, for example jobs with an unchanged etag and filtering these jobs executions in the job overview.

JavaDocs are missing.
README should be extended
Refactor / improve example-jobs

Hello, during opening of the jobdefinitionspage with RC5 I get the following exception:
Whitelabel Error Page

This application has no explicit mapping for /error, so you are seeing this as a fallback.

Mon Feb 13 12:02:10 CET 2017
There was an unexpected error (type=Internal Server Error, status=500).
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.thymeleaf.exceptions.TemplateInputException: Exception parsing document: template="jobdefinitions", line 33 - column 139

The LdapAuthenticationFilter is used to authenticate calls to /internal. This should not only be a feature for Toggles but for all /internal endpoints.

With the current way the URLs are constructed (for example on the internal index page), the href will break if you have a trailing slash and lead to links like ../internal/internal/something

In our current project we have several jobs. One of them runs fairly often (every 10 seconds or so), where as the others run only once per day. In the current job-overview page in the admin area of the edison-microservice it is very hard to see runs of those rarely running jobs, as the list is full with outputs of the one job that runs very often.

The examples should be extracted into a new project 'edison-examples'.

Edison is currently including Thymeleaf 2 (via spring-boot-starter-thymeleaf) and Thymeleaf 3 (direct dependency).

The Default should be Thymeleaf 2 with an optional to manually configure Thymeleaf 3. In Edison 2.0.0, the default will be changed; see #53

JavaDocs are missing.
No README.md

Currently there are NotNull and NotEmpty annotations on the service resp. servers fields. This restriction is not needed if the whole client is not enabled.

It should not be necessary to copy/paste thyme leaf templates in order to customize the nav bar of /internal pages