In my marathon definitions directory, i have a dirty git state right now, because i am preparing a new marathon definition for a new app:

➜  marathon-jobs git:(master) ✗ git st
On branch master
...
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	X.json

nothing added to commit but untracked files present (use "git add" to track)

When i trigger a typical status call the JobLoadException will be thrown and i don't get results at all:

➜  chapi git:(master) bin/chapi status


  [Chapi\Exception\JobLoadException]
  Unable to load json job data from "/my/path/X.json". Please check if the json is valid.


status

Expected results would be that i get a listing of Chronos and Marathon tasks and when i have a dirty state (like X.json) i expect a info or warning note.

I saw this fatal error (on current master)

➜  chapi git:(master) bin/chapi scheduling
PHP Fatal error:  Uncaught TypeError: Argument 1 passed to Chapi\Entity\Chronos\JobStatsEntity::__construct() must be of the type array, object given, called in /Users/andygrunwald/Development/chapi/src/Service/Chronos/JobStatsService.php on line 66 and defined in /Users/andygrunwald/Development/chapi/src/Entity/Chronos/JobStatsEntity.php:28
Stack trace:
#0 /Users/andygrunwald/Development/chapi/src/Service/Chronos/JobStatsService.php(66): Chapi\Entity\Chronos\JobStatsEntity->__construct(Object(stdClass))
#1 /Users/andygrunwald/Development/chapi/src/Commands/SchedulingViewCommand.php(111): Chapi\Service\Chronos\JobStatsService->getJobStats('Bluekai-FillRed...')
#2 /Users/andygrunwald/Development/chapi/src/Commands/SchedulingViewCommand.php(82): Chapi\Commands\SchedulingViewCommand->printTimeLineTable(1481793861, 1481801061)
#3 /Users/andygrunwald/Development/chapi/src/Commands/AbstractCommand.php(78): Chapi\Commands\SchedulingViewCommand->process()
#4 /Users/andygrunwald/Development/chapi/vendor/symfony/console/Command/Command. in /Users/andygrunwald/Development/chapi/src/Entity/Chronos/JobStatsEntity.php on line 28

Fatal error: Uncaught TypeError: Argument 1 passed to Chapi\Entity\Chronos\JobStatsEntity::__construct() must be of the type array, object given, called in /Users/andygrunwald/Development/chapi/src/Service/Chronos/JobStatsService.php on line 66 and defined in /Users/andygrunwald/Development/chapi/src/Entity/Chronos/JobStatsEntity.php on line 28

TypeError: Argument 1 passed to Chapi\Entity\Chronos\JobStatsEntity::__construct() must be of the type array, object given, called in /Users/andygrunwald/Development/chapi/src/Service/Chronos/JobStatsService.php on line 66 in /Users/andygrunwald/Development/chapi/src/Entity/Chronos/JobStatsEntity.php on line 28

Call Stack:
    0.0002     355120   1. {main}() /Users/andygrunwald/Development/chapi/bin/chapi:0
    0.0093    1669400   2. Symfony\Component\Console\Application->run() /Users/andygrunwald/Development/chapi/bin/chapi:12
    0.0113    1965832   3. Symfony\Component\Console\Application->doRun() /Users/andygrunwald/Development/chapi/vendor/symfony/console/Application.php:118
    0.0114    1965832   4. Symfony\Component\Console\Application->doRunCommand() /Users/andygrunwald/Development/chapi/vendor/symfony/console/Application.php:187
    0.0114    1965832   5. Symfony\Component\Console\Command\Command->run() /Users/andygrunwald/Development/chapi/vendor/symfony/console/Application.php:820
    0.0116    1968408   6. Chapi\Commands\AbstractCommand->execute() /Users/andygrunwald/Development/chapi/vendor/symfony/console/Command/Command.php:254
    0.0749    3038280   7. Chapi\Commands\SchedulingViewCommand->process() /Users/andygrunwald/Development/chapi/src/Commands/AbstractCommand.php:78
    0.1031    3999464   8. Chapi\Commands\SchedulingViewCommand->printTimeLineTable() /Users/andygrunwald/Development/chapi/src/Commands/SchedulingViewCommand.php:82
    0.3782    4846032   9. Chapi\Service\Chronos\JobStatsService->getJobStats() /Users/andygrunwald/Development/chapi/src/Commands/SchedulingViewCommand.php:111
    0.4640    4850128  10. Chapi\Entity\Chronos\JobStatsEntity->__construct() /Users/andygrunwald/Development/chapi/src/Service/Chronos/JobStatsService.php:66

Will try to write a unit test and fix it.

I had a job file with a missing "{" at the beginning.
The error message I got was

Argument 1 passed to "Chapi\Entity\Chronos\JobEntity::__construct" must be an array or object

It would be nice if this could be more descriptive and, if possible, point to the file/line producing the error.

Sometimes it's hard to manage upcoming chronos task and to select a good timeframe with less db impact and stuff like this. It would be nice to have a scheduling overview like this example:

JopABC    12.30---<===========12.45=======>-------13.00------
JobXYZ    12.30---------------12.45---<==>--------13.00---<==

Improve epsilon validation to check if epsilon is less than the period of scheduling jobs

It's planed to drop the php 5.4 support to be able to update the external dependencies to the latest versions

When you deploy a docker container as a marathon job with a "servicePort": 0 definition, a random allocated port (from Mesos) will assigned as a servicePort.
With the usage of marathon-lb this is fine, because your random service port will be mapped to a domain in marathon-lb.

When you launch a golang application (without a docker env) in the same way (with "servicePort": 0), the Expected Behavior is the case. There, everything is working.

Expected Behavior

When your job is deployed and you hit chapi status, your job is not our of sync and everything is fine

Changes to be committed
  (use 'chapi reset <job>...' to unstage)


Changes not staged for commit
  (use 'chapi add <job>...' to update what will be committed)
  (use 'chapi checkout <job>...' to discard changes in local repository)

  Updated jobs in local repository:

Current Behavior

When your job is deployed and you hit chapi status, your job always shows as "modified:

Changes to be committed
  (use 'chapi reset <job>...' to unstage)


Changes not staged for commit
  (use 'chapi add <job>...' to update what will be committed)
  (use 'chapi checkout <job>...' to discard changes in local repository)

  Updated jobs in local repository:
	modified marathon job:	/my-service/foo

When you hit chapi diff, the diff shows the random allocated service port as a difference:

$ chapi diff /my-service/foo

diff /my-service/foo
	...
	container:                   "protocol": "tcp",
+	container:                  "servicePort": 0,
-	container:                  "servicePort": 10021,
	container:                   "hostPort": 0,
	...

Possible Solution

Exclude container.docker.portMappings.*.servicePort from diff.

Steps to Reproduce (for bugs)

1. Deploy a Docker Container with "servicePort": 0 on Marathon
2. Hit chapi status, you should see your newly deployed job
3. Hit chapi diff, you should see a diff in your servicePort with a random allocated port

Context

The more jobs you deploy in this way, the more "out of sync" jobs appear.
When we apply an automated check on this, this would always be red / out of sync.

Your Environment

• Chapi version: master (rev: a171472)
• PHP version: PHP 7.0.21 (cli) (built: Jul 29 2017 09:17:51) ( NTS )
• Operating System and version: macOS Sierra 10.12.6 (16G29)
• Mesos version: 1.1.0
• Marathon version: 1.4.1
• Chronos version: 2.4.0

{
  "id": "/my-service/foo",
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "org/container:v1.0",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 0,
          "servicePort": 0,
          "protocol": "tcp"
        }
      ]
    }
  },
  "labels": {
    "HAPROXY_GROUP": "internal",
    "HAPROXY_0_VHOST": "my-domain.mesos-stage.trivago.trv",
  },
  "cpus": 1,
  "mem": 512,
  "instances": 1
}

When showing diffs for Marathon jobs, chapi will show changes in the order of the environment variables, even when nothing about the configuration itself changed. As this seems to be a problem just concerning ordering in the diff and visualization, I think it would be an improvement to have chapi ignore those changes.

Extract api client components to subpackage for single usage

Add a validation for dependency jobs:

• check if the parent job(s) exists
• check if the parent job(s) are in the current commit index
• check for api request order for dependency jobs

Wildcards for identifier would be great:

➜  chapi git:(master) bin/chapi diff hotelmanager-campaignmanager-*
zsh: no matches found: hotelmanager-campaignmanager-*

If the validation of a job failed you get currently no output information for this.
Add a validation output if a job contains invalid parameters.

We runned into an issue with the settings for "uris" and "environmentVariables".
The current method getSimpleArrayCopy() for comparison didn't support more than one level.

{
    "name" : "foo",
    "command" : "bar",
    "schedule" : "R/2016-03-15T11:15:00Z/P1D",
    "description" : "foo bar",
    "scheduleTimeZone":"Europe/Berlin",
    "epsilon" : "PT5M",
    "owner" : "[email protected]",
    "ownerName" : "Paul",
    "async" : false,
    "executor" : "",
    "disabled" : false,
    "softError" : false,
    "cpus" : 0.2,
    "mem" : "128",
    "disk" : "24",
    "highPriority" : false,
    "retries" : 0,
    "uris" : ["http://a.url.com"],
    "environmentVariables" : [
        {
          "name" : "FOO",
          "value" : "bar"
        }
    ]
}

If a job currently running on chronos the DatePeriod comparison for scheduling jobs doesn't work correctly. The jobs will detect as not equal and displayed in diff and status view.

add db support for "local" storage (mysql/mongo) instead of current file based repository and make it configurable

For server installations it's better to send the necessary params via options without the question input

To maintain more than one mesos cluster it would be helpful to provide a "profile" handling like aws-cli do it: http://docs.aws.amazon.com/cli/latest/reference/

The current validator for chronos jobs don't detect issues like this:

{
  "schedule":"R/2017-03-17T08:52:00Z/P1D",
  "scheduleTimeZone":"Europe/Berlin",
  "epsilon":"PT15M",
  "name":"job-name-a",
  "command":"echo jobA",
  "description":"",
  "owner":"",
  "ownerName":"",
  "parents": [
      "job-name-b"
  ],
  "async":false,
  "executor":"",
  "disabled":false,
  "softError":false,
  "cpus":"0.1",
  "mem":"32",
  "disk":"24",
  "highPriority":false,
  "retries": 0
}

Check https://github.com/box-project/amend and implement self-update functionality for phar support

Chronos "don't like it" if you change a scheduling job to a dependency job. In this case it's more safe to delete the job first and add them again.

To be able to move our chronos jobs from cluster A to cluster B smoothly we need a feature to handle two different chronos jobs repositories.

Chapi assumes that the configuration always contains both a marathon and a chronos service. This causes at least two problems in practice:

• It's impossible to set up chapi in an environment where you only have one but not the other

• Even in an environment where both frameworks are used, you may have situations where you'd want to only address one of them. Chapi doesn't provide any command line arguments to select just one.

Setting the config variables to null like README.md suggests worked at some point but doesn't anymore.

I would generalize the configuration mechanism a bit to overcome these problems.

Example configuration:

profiles:
  # Profile for addressing marathon only
  acme-prod-marathon:
    services:
      - framework: marathon
        url: http://marathon.acme.com:8080/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/marathon-jobs"
        
  # Profile for addressing chronos only
  acme-prod-chronos:
    services:
      - framework: chronos-2.0
        url: http://chronos.acme.com:4400/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/chronos-jobs"

  # Profile for addressing both simultaneusly
  acme-prod:
    services:
      - framework: marathon
        url: http://marathon.acme.com:8080/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/marathon-jobs"
      - framework: chronos-2.0
        url: http://chronos.acme.com:4400/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/chronos-jobs"

  # For ease of maintenance, a simple include mechanism could 
  # make the above definition shorter:
  acme-prod:
    services:
      - include: acme-prod-chronos
      - include: acme-prod-marathon

  # With the assumption gone that there are always exactly one marathon
  # and chronos service, we could support a fictional use case like:
  acme-farm:
    services:
      - framework: marathon
        url: http://marathon-ber.acme.com:8080/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/marathon-jobs"
        ignore:
          - "!*-ber"
      - framework: marathon
        url: http://marathon-lon.acme.com:8080/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/marathon-jobs"
        ignore:
          - "!*-lon"
      - framework: marathon
        url: http://marathon-nyc.acme.com:8080/
        http_username: acme
        http_password: acme123
        repository_dir: "/Users/jsmith/src/marathon-jobs"
        ignore:
          - "!*-nyc"

With this pattern, you could simply pass a profile name on the command line to select the environment(s) you want chapi to target.

Also it would be possible to add support for other mesos frameworks (or multiple flavors / versions of current the frameworks), hence "chronos-2.0" in the example. ("Framework" might not be the best wording for this though, since its's a Mesos-specific term, unless we intend chapi to manage Mesos-based services only)

Chapi need adjustments for the breaking changes of the chronos 3.0.0 release

Providing deleting all tasks for a Job by jobname

https://mesos.github.io/chronos/docs/api.html#deleting-all-tasks-for-a-job

Boolean properties like "disabled" will not compare correctly

To avoid a wrong job status after a commit it's better to invalid the job status cache

When you deploy services on Marathon and kill them later, the namespace / group will stay.
If no app will be deployed in this group later on, this group will stay empty forever.

See

Current Behavior

After time your Marathon instance will have several empty groups in the UI / API.

Possible Solution

Provide Framework (Marathon / Chronos) specific commands.
For this request it would be the /v2/groups/{group_id}/ DELETE command descibed in https://mesosphere.github.io/marathon/api-console/index.html

So the descriptive workflow would be

• Get all groups from the server
• Check if they are empty
• When they are empty, delete the group

Steps to Reproduce (for bugs)

1. Deploy a service on Marathon with ID /foo/my-service
2. Delete the service from marathon
3. The empty group /foo will stay

Your Environment

• Chapi version: master (Rev: c683314)
• PHP version: v7.1.11
• Operating System and version: Mac OSx
• Mesos version: v1.1.0
• Marathon version: v 1.4.1
• Chronos version: -

With the marathon support we recognised that a "ignore job" handling would be useful.

Validate job names for invalid characters. Example: "job-name:do:something"

Some commands like commit or status need ~3sec. We need to measure the current bottlenecks and try to improve the performance for big job repositories with > 200 jobs.

Currently chapi don't recognize that a job name is already used in another job file. In the end one job definition overwrite the other one. We need some kind of validation for unique job names.

Since most of the configuration on chronos is pretty similar to marathon, it would make sense to use the similar components for both systems. Then, the configuration, validation and push the configuration to marathon would be simple as well.

Add a separate command to validate the local job repository.

Add more info massages for commit and local job updates