Use regexp [A-Za-z0-9_\-]+@[A-Za-z0-9_\-]+:[0-9]+ to validate parsed string.

if you run the example framework and executor and pass the -task-count=2 value, mesos ends up running 3 tasks instead of two.

I see this by looking at the logs and I see:

...Prepared task: go-task-1...
...Prepared task: go-task-2...
...Prepared task: go-task-3...

The error seems on this file
https://github.com/mesos/mesos-go/blob/master/examples/test_framework.go#L109

for sched.tasksLaunched <= sched.totalTasks &&

if I change it to

for sched.tasksLaunched < sched.totalTasks &&

then it only schedules the number of tasks I passed on the command line.

If you agree this is an issue, I can send a quick PR to fix it.

Thanks

• Add documentation for binding
• Add CI link (drone.io)
• Add to godoc site

Compilation seems to assume I have mesos headers installed in /usr/local/include, is it possible to compile with mesos installed elsewhere? This may be more of a Go problem than a mesos-go one, but the folks in #mesos asked me to make a ticket for now.

$ go get github.com/mesosphere/mesos-go/example_framework
# github.com/mesosphere/mesos-go/mesos
.go/src/github.com/mesosphere/mesos-go/mesos/executor_driver.cpp:5:30: fatal error: mesos/executor.hpp: No such file or directory
 #include <mesos/executor.hpp>
                              ^
compilation terminated.

Currently, the health check will going concurrently with other interfaces (e.g. launchTask, killTask). That makes us to use locks to avoid race, which is ugly and error prone IMO
Maybe we can move this part of logic into the messenger, that can give us a more "libprocess like" messenger.

But we need to make sure on the scheduler's side, the health check stuff is also needed.

Update:
The scheduler side will have a master detector, which also needs to be serialized with other interfaces. The detector and the slave health check is pretty different.

Consider another alternative: Creating an event loop in the executor driver, and have all messages and health check events go through that loop.

I'm getting an error while trying to follow build instructions.

root@441a0c5a33df:/go/mesos-go/examples# go get github.com/golang/glog github.com/mesos/mesos-go/mesosproto github.com/mesos/mesos-go/mesosutil github.com/mesos/mesos-go/scheduler
# github.com/mesos/mesos-go/mesosproto
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:661: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:720: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:737: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:754: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:771: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:789: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:806: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:833: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:860: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:970: undefined: proto.ErrWrongType
/go/src/github.com/mesos/mesos-go/mesosproto/log.pb.go:970: too many errors

Any ideas?

see #53 (comment)

Related to #49. This is to track integration with the scheduler component.

TODO

• Initial integration work
• Add leader detection tests using zk-detector

I'm trying to to follow README directions, and running
go get github.com/mesosphere/mesos-go/example_executor, however I'm getting the following error

# github.com/mesosphere/mesos-go/mesos
src/github.com/mesosphere/mesos-go/mesos/executor_driver.cpp:5:30: fatal error: mesos/executor.hpp: No such file or directory
compilation terminated.

I'm not seeing mesos/executor.hpp file in the repo.

Is it a bug?

Revisit logging to make more consistent.

• Standardize on levels for each type of message
• Ensure consistent logging levels
• Make use of the verbosity(level) where to control app verbosity

When testing the test headers:

In file included from test_headers.cpp:1:0:
/usr/local/include/mesos/mesos.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/usr/local/include/mesos/mesos.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/usr/local/include/mesos/mesos.pb.h:14:2: error: #error your headers.
#error your headers.
^
In file included from test_headers.cpp:1:0:
/usr/local/include/mesos/mesos.pb.h:26:55: fatal error: google/protobuf/generated_enum_reflection.h: No such file or directory
#include <google/protobuf/generated_enum_reflection.h>
^
compilation terminated.

This is follow-up from #46.

This is not true, 0.0.0.0 is "all ips", not localhost. Because of this mesos cannot reach back to scheduler and framework cannot register itself. This probably works on localhost, but does not work with 3 masters on different hosts.

I think we should pass upid.UPID into scheduler.NewMesosSchedulerDriver to make it more flexible.

Which is compatible to goprotobuf, but much faster.
https://code.google.com/p/gogoprotobuf/

Also consider to pull the protobuf out of mesos package...

Could we leverage this?

https://github.com/youtube/vitess/tree/master/go/rpcwrap/auth

How do you define a dynamic command in mesos.taskInfo?

I'm trying to change a definition mesos.TasksInfo from using an executor, to executing any command.

I see the command being executed but getting a SegFault right away.

&mesos.TaskInfo{
                Name: proto.String("go-task"),
                TaskId: &mesos.TaskID{
                    Value: proto.String("go-task-" + strconv.Itoa(s.taskId)),
                },
                SlaveId:  offer.SlaveId,
                //Executor: executor,
                Command: &mesos.CommandInfo{
                    Value: proto.String("echo $((1+1))"),
                },
                Resources: []*mesos.Resource{
                    mesos.ScalarResource("cpus", 1),
                    mesos.ScalarResource("mem", 512),
                },
            },

Any tips?

Will do this after merging the refactor branch into master.

Since there many components shared between scheduler/executor, create new common package to host common logic such as transporter, messenger, constant values, etc.

Add the ability to specify the hostname and port the Scheduler driver should bind and advertise to the master. See #52 for backstory.

I want to use Sourcegraph code search with mesos-go. A project maintainer needs to enable it to set up a webhook so the code is up-to-date there.

Could you please enable mesos-go on @sourcegraph by going to https://sourcegraph.com/github.com/mesos/mesos-go and clicking on Settings? (It should only take 15 seconds.)

It really helps when I am navigating source and would be highly appreciated.

After looking at #12 and #40 it seems obvious that build instructions are needed.

I personally got this:

# github.com/mesos/mesos-go/mesos
In file included from src/github.com/mesos/mesos-go/mesos/executor_driver.cpp:5:
In file included from /usr/local/include/mesos/executor.hpp:26:
In file included from /usr/local/include/mesos/mesos.hpp:22:
/usr/local/include/mesos/mesos.pb.h:17:2: error: This file was generated by an older version of protoc which is
/usr/local/include/mesos/mesos.pb.h:18:2: error: incompatible with your Protocol Buffer headers.  Please
/usr/local/include/mesos/mesos.pb.h:19:2: error: regenerate this file with a newer version of protoc.

And after installing mesos after protobuf (homebrew on mac) into this:

# github.com/mesos/mesos-go/mesos
In file included from src/github.com/mesos/mesos-go/mesos/executor_driver.cpp:5:
In file included from /usr/local/include/mesos/executor.hpp:26:
In file included from /usr/local/include/mesos/mesos.hpp:22:
/usr/local/include/mesos/mesos.pb.h:9:10: fatal error: 'google/protobuf/stubs/common.h' file not found

Real fix for the whole thing is moving away from linking to libmesos and using http api, like volt framework (https://github.com/VoltFramework/volt) does. Should I file an issue for that as well?

Move Auth/SASL support under available.

Change var foo = bar to foo := bar.
And other minus style issues.

See PR #80

Add logic to schedulerdriver.StatusUpdate() to

• Validate connection state
• Check before sending ACK to Master

Hi,
I'm wondering if it'd be possible to also implement the Slave interface in mesos-go, so we could build a pure slave in Go without any dependency on libmesos.

Would that work?

Messenger.Send() will have a data race with Messenger.Start() on the upid.
Should try to make sure the Send() won't be called before the Messenger completely started.

how do i get the task output after a task has completed?

When master is provided in the form of master@:, scheduler.NewSchedulerDriver fails.

Instead of burying flags inside the bindings library, we should leave that to the actual framework schedulers and executors. Instead, modify the driver constructors to accept config structs. This is way more flexible and allows for easier testing and experimentation with other types of transports, for example TLS.

For example:

package scheduler

type DriverConfig {
  NewMessenger func() (messenger.Messenger,error)
  AuthProvider string
  Scheduler Scheduler
  Framework *mesos.FrameworkInfo
  Master string
  Credential *mesos.Credential
}

func NewMesosSchedulerDriver(config DriverConfig) (*MesosSchedulerDriver, error) {
  // ...
  driver.messenger, err = config.NewMessenger()
  //...
}

package messenger

func NewHttpFactory(host string, port int) (func() (Messenger, error)) {
  return func() (Messenger, error) {
    upid := &upid.UPID{
        ID:   "scheduler(1)",
        Host: host,
        Port: strconv.Itoa(port),
    }
    return NewHttp(upid), nil
  }
}

package main

import (
  "flag"
  //...
)

var (
  hostFlag := flag.String(...)
  portFlag := flag.Int(...)
  authProviderFlag := flag.String(...)
)

// run some framework
func main() {
  config := DriverConfig{
    NewMessenger: messenger.NewHttpFactory(*hostFlag, *portFlag),
    AuthProvider: *authProviderFlag,
    // ... other config here
  }
  driver, err := scheduler.NewMesosSchedulerDriver(config)
  //...
}

http://tip.golang.org/cmd/go/#hdr-Generate_Go_files_by_processing_source

Keep global scheduler and executor counters used for generation of UPID (i.e. scheduler(1), scheduler(2), ... , scheduler(n))

Hi,

I just pulled master, and saw that my executor doesn't start up anymore. This executor used to startup until a few weeks back.

E0208 02:08:22.506744 15178 http_transporter.go:128] HTTPTransporter failed to listen: listen tcp4: unknown port tcp4/

E0208 02:08:22.507236 15178 messenger.go:150] Failed to start messenger: listen tcp4: unknown port tcp4/

E0208 02:08:22.507247 15178 executor.go:388] Failed to start executor: listen tcp4: unknown port tcp4/

@vladimirvivien Do you have any suggestions?

For now, I only verify that the messages are passed correctly, and the handlers are called.
We should try to verify the content of the messages later.

Add a mock master detector to allow testing outside of the detector package.

I wrote a custom executor with the new "pure" Go bindings. And I am observing the cpu utilization of the executor process to be always ~100% with load average of the server over or close to 1.00

Can you guys test whether it's the same case with your executors too?

#53 (comment)

Current ExecDriver API depends on solely on ENV variables to access execdriver config values. I think that should, at most, be optional. The primary way to pass value when using the API should be via function params (ExecDriver constructor function for instance).

So this refactor will do the followings

• Modify the constructor method to accept all necessary params
• Passed params take precedence
• If value is not passed, then the code will cascade search for missing values from ENV
• If required value not present as param nor as ENV, then error out.

I have a toy framework that attempts re-registration (but uses a 20sec failover timeout). If I attempt to re-register once the frameworkID has been marked as completed, the following infinite loop occurs but my program does not terminate:

I0218 21:17:06.968481   23437 main.go:220] Attempting to resume with FrameworkID = 20150219-031126-177048842-5050-1714-0000
I0218 21:17:06.968760   23437 main.go:223] &FrameworkInfo{User:*,Name:*Test Framework (Go),Id:&FrameworkID{Value:*20150219-031126-177048842-5050-1714-0000,XXX_unrecognized:[],},FailoverTimeout:*20,Checkpoint:*true,Role:nil,Hostname:nil,Principal:nil,WebuiUrl:nil,XXX_unrecognized:[],}
I0218 21:17:06.970400   23437 scheduler.go:179] Initializing mesos scheduler driver
I0218 21:17:06.970699   23437 scheduler.go:449] Starting the scheduler driver...
I0218 21:17:07.971133   23437 scheduler.go:490] Mesos scheduler driver started with PID= scheduler(1)@10.141.141.1:33160
I0218 21:17:07.971243   23437 scheduler.go:597] Scheduler driver running.  Waiting to be stopped.
I0218 21:17:07.978864   23437 scheduler.go:886] Aborting driver, got error ' Completed framework attempted to re-register '
I0218 21:17:07.978904   23437 scheduler.go:649] Aborting framework [20150219-031126-177048842-5050-1714-0000]
I0218 21:17:07.978928   23437 scheduler.go:655] Ignoring Abort, master is disconnected.
I0218 21:17:07.978959   23437 main.go:187] Scheduler received error: Completed framework attempted to re-register
I0218 21:17:09.189569   23437 scheduler.go:886] Aborting driver, got error ' Completed framework attempted to re-register '
I0218 21:17:09.189602   23437 scheduler.go:649] Aborting framework [20150219-031126-177048842-5050-1714-0000]
I0218 21:17:09.189616   23437 scheduler.go:655] Ignoring Abort, master is disconnected.
...forever...

The key here is this message "Ignoring Abort, master is disconnected". In situ:

func (driver *MesosSchedulerDriver) Abort() (mesos.Status, error) {
        ...(snip)...
    if !driver.connected {
        log.Infoln("Ignoring Abort, master is disconnected.")
        return driver.Status(), fmt.Errorf("Unable to Abort, driver not connected.")
    }
    _, err := driver.Stop(true)
    stat := mesos.Status_DRIVER_ABORTED
    driver.setStatus(stat)
    return stat, err
}

I don't think I understand the point of the if-block in this code. Any abort should probably just kill the driver, no?

tried to run a http server in the background alongside the driver (separate go routine) but port 80 seems to be in use???

Hi,

I am trying to mesos-go on mac os x, and I got an error that I don't libmesos.so, I changed it to check if I have dylib and I am getting the next error -

λ ~/code/mesos/mesos-go/ master* make
go get code.google.com/p/goprotobuf/proto
go get code.google.com/p/goprotobuf/protoc-gen-go
cd c-bridge; make all
ar -rcs libcbridge.a executor_driver.o scheduler_driver.o
go install mesos.apache.org/example_framework
can't load package: package mesos.apache.org/example_framework: cannot find package "mesos.apache.org/example_framework" in any of:
    /usr/local/Cellar/go/1.2/libexec/src/pkg/mesos.apache.org/example_framework (from $GOROOT)
    /Users/yosy/code/go/src/mesos.apache.org/example_framework (from $GOPATH)
make: *** [examples] Error 1

What should I do in order to make it work on Mac OS X? or this project is compatible only with linux?