Git Product home page Git Product logo

kerub's Introduction

Coverage Reliability Rating Security Rating Technical Debt Vulnerabilities Duplicated Lines (%) Lines of Code FOSSA Status

FreeBSD CentOs Ubuntu OpenSUSE

kerub

Logo

A lightweight prototype IaaS application

Objective

The objective of this project is to demonstrate some technologies and solutions in the IaaS field. Such solutions are:

  • User expectations - the user sets the service level (SLA) of the virtual resources by adding expectations
  • Scheduling as an optimization/constraint enforcement problem - which means SLA defined by user is checked on each event and the required steps are taken to maintain it
  • No host-cluster - this construct was created only to help human-operators to do scheduling - a bad use of payed work-hours

Architectural shift:

  • No host agent
  • Event driven, no poll
  • NoSQL document store for VM data
  • Simplicity in deployment and development
  • Minimal hardware requirements

Put rare features to use:

  • KSM

Despite all of the above objectives:

  • it is not for architecture astronauts
  • not a playground for design patterns
  • must be simple and readable
  • must be tested and working well

Status

Kerub is in research/development phase, however you should be able to run and use some minimalistic functionality:

  • storage management with LVM, GVinum (FreeBSD) and filesystems
  • storage protocols like nfs and iscsi
  • KVM hypervisor
  • SPICE virtual console

Tests

Some tests are included in the application, both unit and integration tests, these tests can simulate workload and run very quick (the full build is around 5 minutes on a dual-core i7 CPU). However these tests can not cover all aspects, for example we can not have a real host with an actual operating system installed on it, the command executions are faked, etc.

To fill this gap, there is another package only for tests called kerub-ext-tests. This however runs against packaged (RPM, DEB) versions of the application, sophisticated to setup, requires hypervisor hardware and software and a full run takes several hours.

There is a second separate test package for microbenchmarks: kerub-microbenchmarx This is a JMH-based benchmark collection to help tuning (mostly but only) the planner.

How to get started

git clone https://github.com/kerubistan/kerub/
cd kerub
mvn jetty:run

Open http://localhost:8080/ in your favourite browser, and you have a controller running.

First of all, you will need a few (at least one) host to work with. This can be a virtual machine in test environments if nested virtualization is enabled, or a physical server. The host needs a few software to work with:

  • an operating system (see above)
  • all hosts need a working ssh daemon, root must be able to connect remotely with public key authentication
  • storage software, like lvm, zfs or gvinum
  • storage protocols, like iscsi
  • virtualization software, like kvm+qemu with libvirt (virtualbox, xen and others under development)
  • monitoring and hardware discovery software, typically the ones found in any linux distribution

Kerub will find whatever is installed.

OS Packages

OS / Distribution packaging project status
Fedora kerub-fedora maintained
CentOS kerub-centos maintained
ubuntu kerub-ubuntu development
openSUSE kerub-openuse maintained

License

FOSSA Status

kerub's People

Contributors

dependabot[bot] avatar fossabot avatar k0zka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

kerub's Issues

host details view

A host details dialog with

  • Static data of the host
  • the dynamic data of the host changing in time

authorization

protect the rest services with authorization for roles and by ownership of objects

host detection issue with centos 6.6

Looks like the /etc/os-release file is not present, but the host detection should tolerate this

Caused by: org.apache.sshd.common.SshException: SFTP error (2): No such file
    at org.apache.sshd.client.sftp.DefaultSftpClient.checkHandle(DefaultSftpClient.java:310) ~[sshd-core-0.12.0.jar:0.12.0]
    at org.apache.sshd.client.sftp.DefaultSftpClient.open(DefaultSftpClient.java:425) ~[sshd-core-0.12.0.jar:0.12.0]
    at org.apache.sshd.client.sftp.DefaultSftpClient$4.<init>(DefaultSftpClient.java:646) ~[sshd-core-0.12.0.jar:0.12.0]
    at org.apache.sshd.client.sftp.DefaultSftpClient.read(DefaultSftpClient.java:642) ~[sshd-core-0.12.0.jar:0.12.0]
    at com.github.K0zka.kerub.host.HostPackage$SshClientUtils$e871759f.getFileContents(SshClientUtils.kt:111) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostPackage$SshClientUtils$e871759f$getFileContents$1.invoke(SshClientUtils.kt:102) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostPackage$SshClientUtils$e871759f$getFileContents$1.invoke(SshClientUtils.kt) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostPackage$SshClientUtils$e871759f.use(SshClientUtils.kt:84) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostPackage$SshClientUtils$e871759f.getFileContents(SshClientUtils.kt:101) ~[classes/:na]
    at com.github.K0zka.kerub.host.distros.LsbDistribution.readLsbReleaseProperties(LsbDistribution.kt:39) ~[classes/:na]
    at com.github.K0zka.kerub.host.distros.LsbDistribution.detect(LsbDistribution.kt:31) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostCapabilitiesDiscoverer.detectDistro(HostCapabilitiesDiscoverer.kt:95) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostCapabilitiesDiscoverer.discoverHost(HostCapabilitiesDiscoverer.kt:36) ~[classes/:na]
    at com.github.K0zka.kerub.host.HostCapabilitiesDiscoverer.discoverHost$default(HostCapabilitiesDiscoverer.kt:34) ~[classes/:na]

fix swagger

re-introduce readable API documentation without the server-side jaxrs swagger plugin

flyweight the data

When loading data to memory either from hard disk, or before actually saving it to memory (and disk) it could be de-duplicated to save memory

websocket faliure is not logged

Web socket message failures (including deserialization, processing, response, etc) are not logged by spring framework. It should be captured and logged or if possible, handled.

ensure unique host properties

When joining a host, ensure that some unique properties of the host are indeed unique in the installation:

  • host address
  • system uuid (from dmidecode, if any info is provided)

implement paging

At the moment all entities are listed by the service without paging and sorting.

It does not look good, it does not perform good.

audit

  • create audit AOP interceptor
  • insert/update/delete should be 3 different types (right now AuditEntry is the only one)
  • audit must be configured on selected daos:
    • host
    • vm
  • Audit datastore persistent, write-optimized (not much entries kept in memory)

clean up the kotlin compiler warnings

There are 13 compiler warnings at the moment, it would be nice to have 0

/home/kocka/sources/kerub/src/test/kotlin/com/github/K0zka/kerub/host/ControllerManagerImplTest.kt
Warning:(34, 9) Kotlin: Variable 'controllerDynamic' is never used
/home/kocka/sources/kerub/src/test/kotlin/com/github/K0zka/kerub/host/HostCapabilitiesDiscovererTest.kt
Warning:(98, 45) Kotlin: Elvis operator (?:) always returns the left operand of non-nullable type kotlin.Long
Warning:(98, 53) Kotlin: Unnecessary safe call on a non-null receiver of type kotlin.String
Warning:(98, 63) Kotlin: Unnecessary safe call on a non-null receiver of type kotlin.Int
Warning:(153, 42) Kotlin: This syntax for lambda is deprecated. Use short lambda notation {a[: Int], b[: String] -> ...} or function expression instead.
Warning:(175, 7) Kotlin: Variable 'host' is never used
Warning:(181, 48) Kotlin: Unnecessary non-null assertion (!!) on a non-null receiver of type com.github.K0zka.kerub.host.HostCapabilitiesDiscoverer
Warning:(183, 40) Kotlin: Unnecessary safe call on a non-null receiver of type com.github.K0zka.kerub.model.HostCapabilities
Warning:(184, 45) Kotlin: Unnecessary safe call on a non-null receiver of type com.github.K0zka.kerub.model.HostCapabilities
/home/kocka/sources/kerub/src/test/kotlin/com/github/K0zka/kerub/host/SshClientUtilsTest.kt
Warning:(54, 37) Kotlin: This syntax for lambda is deprecated. Use short lambda notation {a[: Int], b[: String] -> ...} or function expression instead.
Warning:(64, 33) Kotlin: This syntax for lambda is deprecated. Use short lambda notation {a[: Int], b[: String] -> ...} or function expression instead.
/home/kocka/sources/kerub/src/test/kotlin/com/github/K0zka/kerub/hypervisor/kvm/UtilsTest.kt
Warning:(25, 7) Kotlin: Variable 'dom' is never used
/home/kocka/sources/kerub/src/test/kotlin/com/github/K0zka/kerub/utils/junix/dmi/DmiDecoderTest.kt
Warning:(127, 7) Kotlin: Variable 'devices' is never used

sockJS integration

SockJS is not working, apparently conflicts with the spring configuration.

new/edit VM dialog

A "New VM" dialog to enter details of the VM:

  • VM name
  • Memory size
  • CPU
  • All kind of VM expectations

planner design and implementation

schedule a VM on the assigned hosts based on

  • host static data (hardware info)
  • host dynamic data and history (load info)
  • vm expectations

Some benchmark needed here to see how it performs with a big number of hosts and VM's scheduled

use jsonviews to filter some of the properties

Some properties of the beans are not always useful. For example

  • Host
    • capabilities.packages: not needed when just listing the hosts
  • VM
    • expectations not needed when just listing the vm's

There should be two views defined:

  • Full: all properties
  • List: selected properties

Links:

cryptic angular error at initialization

Error: e is null
.after/<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:153:309
r@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:7:288
.after@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:153:286
S.prototype[c]@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:155:169
Ld</this.$get</<.enter@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:157:48
td</<.compile/</</</a<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:197:246
B/<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:46:367
M/<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:48:217
w@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:52:29
td</<.compile/</</<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:197:226
p/g.success/<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:72:199
De/e/l.promise.then/J@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:101:87
De/e/l.promise.then/J@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:101:87
De/f/<.then/<@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:102:259
Yd/this.$get</h.prototype.$eval@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:113:28
Yd/this.$get</h.prototype.$digest@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:110:109
Yd/this.$get</h.prototype.$apply@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:113:360
m@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:72:452
w@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:77:463
ye/</B.onreadystatechange@http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js:79:24

http://localhost:8080/webjars/angularjs/1.2.28/angular.min.js
Line 92

fill host capabilities with detected data

when performing host discovery, check if dmidecode is installed, if installed, run it and get all data
if not installed and the host is dedicated, install it first
if not installed and the host is not dedicated, leave the records empty

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.