harvey-os / harvey Goto Github PK
View Code? Open in Web Editor NEWA distributed operating system
Home Page: https://harvey-os.org/
License: MIT License
A distributed operating system
Home Page: https://harvey-os.org/
License: MIT License
This is a re-release of the final version of the 4th Edition of Plan 9 from Bell Labs distributed directly by Bell Labs. 4th Edition was originally released by Lucent Technologies in 2002. It was released under the terms the Lucent Public License Version 1.02. 4th Edition continues the development of the same ideas, but in many ways represents the base of what most people think of as "modern" Plan 9 today. The ubiquitous file protocol, 9P, got its first revision, to 9P2000, which caused changes to many parts of the system. It also saw the debut of venti(8) and fossil(4), which began replacing the venerable old file server, fs(4). Authentication largely moved to a separate agent, factotum(4). Updates moved to replica, and the wiki was created. The original release notes for this edition can be found at /sys/doc/release4.ms. While this release is a recognizable ancestor of the latest Plan 9 available, it is still quite old, and this release is mostly for educational and research purposes. The file protocol and authentication mechanisms are continued in modern Plan 9, though, so researchers looking to run this should have an easier time than with all previous releases. This is a second release of the 4th Edition, marking the final state of Plan 9 as distributed directly by Bell Labs. The only changes between this release and the original 4th Edition release are the new NOTICE, LICENSE, and README files, and moving the old ones. In particular, any copyright notices attributing things to Lucent Technologies or Bell Labs, or notices on restrictions of their use, should be read as historical; all such rights have been transferred to the Plan 9 Foundation and released as open source. Parts (notably, the fonts) maintain trademark and copyright to their respective parties (such as Bigelow & Holmes Inc.) and were included in Plan 9 by permission; those copyrights are unaffected by the transfer to Plan 9 Foundation and remain in effect per the terms found in this distribution.
build.json includes /util/nvram to ease integration of factotum during development.
However, we don't want every Harvey node in the world to have the same hostowner and password.
This is a reminder until Harvey could be distributed with an install program which sets up nvram.
refer to this bug when you push devmnt performance changes.
Given we want to replace libsec, we need a modern well established crypto library that will replace it.
Since it's required by the kernel (random number generation, devtls and so on), it's going to be part of our core system.
Thus we need
To minimize the risk of introducing bugs in the code we get, we adopt a convention based, automated process:
upstream/
, port/*
and any further temporary item requiredsha256sum
against a known valid checksumupstream/
folder (if not already present)Then in each custom project we can do the automatic port (carefully using tools and semantics available on Harvey user space).
Any header added to /sys/include during the process must be added to $HARVEY/.gitignore
and conforms to the Harvey conventions (that is, it should not contains macro guards or #ifdef unrelated to Harvey)
This way we
NOTE
This process is specifically designed for cryptographic tools that are going to be part of the core native system. It does not impose or propose any rule or constraint on other kind of native or glibc based port.
/path/to/fs.c: In function ‘keylist’:
/path/to/fs.c:471:14: warning: unused variable ‘zero’ [-Wunused-variable]
static char zero[Nearend];
fs.c: In function ‘keylist’:
fs.c:471:14: warning: unused variable ‘zero’ [-Wunused-variable]
static char zero[Nearend];
This is quite annoying when there are multiple fs.c files that you have to locate and try out. I discovered that GCC uses the path that is given as input, see https://stackoverflow.com/questions/1282392/can-gcc-be-configured-to-not-print-out-a-full-path-in-warning-error-messages
The solution would be to resolve the complete path of the compiled source file in the build program.
Due to a varargs issue, run/runv functions can't work as usual.
Among other things, we must use shell (rc) to prepare disk for botting the system, doing various part of the work that were done by bootloader in Plan 9.
we have too many clickthroughs for my test. The 'how to get prereqs' is a clickthrough to like 12 lines of text Why the indirection?
Could we drop it in favor of libssl?
Could be a wrapper to libssl?
Do we really need devssl/tls in kernel?
Can't find #P/realmodemem
When aborting 9p requests we seem to lose track of fids, such that the server thinks they are in use and the kernel does not.
This issue is open to note here all the things we are finding about problems running Harvey in Qemu, hacks made for this or broken things.
Delay in boot:
https://review.gerrithub.io/#/c/259279/
main.c:178
"// FIX QEMU. extern int64_t hz;
int64_t hz;"
The ramdisk in plan 9, which runs in user mode, is a dog. It has a major impact on the performance of go test. We need a kernel ramdisk.
As part of the move to the new BUILD system we need to install the BUILD files in the right place.
travis-ci by default provides GCC 4.6 that does not support -Wmaybe-uninitialized
. However, we always use such flags with -Wall
, so that it becomes redundant in gcc versions that support it.
So we should
-Wmaybe-uninitialized
whenever used with -Wall
refers to envsetup, which is gone.
The current implementation of sqrt.s segfaults.
Since we have a C portable version that works, we temporary remove sqrt.s, to avoid duplicated definitions.
In the long term however, we have to fix sqrt.s since amd64 provides a dedicated instruction.
Harvey hasn't any cons available before calling factotum for authentication.
2016/02/19 16:20:34 /usr/bin/ld -o nfs nfs.o -static -e_main -lavl -lcomplete -lcontrol -ldisk -lflate -lframe -lgeometry -lhttpd -lbin -llex -lmemdraw -lmemlayer -lmemdraw -ldraw -lplumb -lregexp -lstdio -lString -lsunrpc -loventi -l9p -lauth -lauthsrv -lndb -lip -lventi -lsec -lauth -lmp -lthread -lmach -lbio -lc -L /home/ubuntu64/harvey/amd64/lib
nfs.o:在函数‘mountMnt’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:130:对‘nfs3Errstr’未定义的引用
nfs.o:在函数‘nfsGetattr’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:191:对‘nfs3Errstr’未定义的引用
nfs.o:在函数‘nfsAccess’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:217:对‘nfs3Errstr’未定义的引用
nfs.o:在函数‘nfsMkdir’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:252:对‘nfs3Errstr’未定义的引用
nfs.o:在函数‘nfsCreate’中:
/home/ubutu64/harvey/sys/src/cmd/nfs.c:291:对‘nfs3Errstr’未定义的引用
nfs.o:/home/ubuntu64/harvey/sys/src/cmd/nfs.c:327: more undefined references to `nfs3Errstr' follow
nfs.o:在函数‘nfsReadDirPlus’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:544:对‘nfs3EntryPlusUnpack’未定义的引用
nfs.o:在函数‘nfsReadDir’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:590:对‘nfs3Errstr’未定义的引用
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:597:对‘nfs3EntryUnpack’未定义的引用
nfs.o:在函数‘threadmain’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:1576:对‘nfs3Prog’未定义的引用
nfs.o:在函数‘dialproc’中:
/home/ubuntu64/harvey/sys/src/cmd/nfs.c:1643:对‘nfs3Prog’未定义的引用
2016/02/19 16:20:34 exit status 1
We need a minimal set of simple and powerful syscalls that can be considered stable.
Thus we have to replace syscalls that can be expressed in term of the others (eg sleep
and alarm
can be moved to user space using awake
) and decide what to do with those syscalls that we inherited but are not currently available in any C header (maybe some are going to be removed and others to be improved).
It's not handling fork well for some reason.This could be a word size issue again.
Running mahjongg in a rio window should result in the mahjongg game being displayed.
rio freezes and the following error message is displayed, both over the rio screen and in the QEMU window:
read fault fail, no segment, pid 82 addr 0x0 pc 0xfffffffff0150365
qlock with ilockdepth 1, stack: k:1df7a3 k:1da556 k:155220 k:1545de k:154382 k:138f90 k:20a39d k:20a1fb k:20afeb k:20b999 k:1f5755 k:14114b k:140ac5 k:1e76b2 k:1e78f3 k:152b4d k:139695
panic: cpu0: cpu0: ilockdepth 1, last lock 0x0 at 0x0, sched called from 0xfffffffff014a1b5
As above
We should remove all code that we are not going to maintain.
For example, are we going to include abaco in Harvey? What about libhtml?
Hi,
I tried to push to gerrit, not a particularly complicated change, I just fixed some warnings, but it keeps getting rejected with:
! [remote rejected] HEAD -> refs/for/master (not Signed-off-by author/committer/uploader in commit message footer)
What am I missing?
Keith
Since we use json files to guide the build, we should give them a bit more love:
..
inclusions with absolute paths (/
is the $HARVEY
dir): so we can move folders easilyThese two scripts need awk and we've worked around it by commenting awk out.
Let's get an awk we can use.
stats does not fail when stats window clicked on with right mouse button
stats quits and displays something like the following in the rio window
stats 90: suicide: sys: trap: fault write addr=0x41fd60 pc=0x4169fc
Approach 1:
Approach 2
init: starting /bin/rc
10.0.2.15 Feb 26 18:44:16 starting dns resolver on 10.0.2.15's /net
machp access spllo, stack: k:13f4cf k:138864 k:19cb35 k:11a7d3 k:1de3d7 k:1de621 k:148db8 k:139695
vbecall: sizeof u is 76
vbecall: sizeof u is 76
vbecall: sizeof u is 76
machp access spllo, stack: k:13f4cf k:138864 k:142638 k:1221f5 k:163892 k:121402 k:11b09c k:11b3be k:1de7b3 k:1de9f4 k:148db8 k:139695
machp access spllo, stack: k:13f4cf k:138864 k:19cabf k:19c82b k:121380 k:11ae09 k:11b3be k:1de7b3 k:1de9f4 k:148db8 k:139695
machp access spllo, stack: k:13f4cf k:138864 k:19caea k:1a4b00 k:1a464f k:1de7b3 k:1de9f4 k:148db8 k:139695
machp access spllo, stack: k:13f4cf k:138864 k:1a48c5 k:1de7b3 k:1de9f4 k:148db8 k:139695
machp access spllo, stack: k:13f4cf k:138864 k:1a4a61 k:1a49dc k:1de7b3 k:1de9f4 k:148db8 k:139695
machp access spllo, stack: k:13f4cf k:138864 k:13d69f k:194f81 k:193dc7 k:1de3d7 k:1de621 k:148db8 k:139695
write fault fail Load(r-x) pid 90 addr 0x41fd60 pc 0x4169fc
Given we want to replace libsec, we need a modern well established crypto library that will replace it.
Since it's required by the kernel (random number generation, devtls and so on), it's going to be part of our core system.
Thus we need
All this should be achieved
git bisect
To this aim we decided to code a set of Go utilities that can automatically vendor a source archive according to directives described in a json file (from download to checksum, including a final git commit
with a standardized message) and automatically verify that the code imported matches the one contained in the archive declared in the json,
This tool would prevent the first obvious threat described above and will reduce the risk of the second, as it is pointless to discuss an automated commit that builds, passes the regression tests and pass the source verification: the only check that remains to reviewers (on this regard) is to ensure that the URI of the archive prescribed in the json and its digests actually match the intended one.
To address the third possible threat in an (almost) trust-less fashion, we need to be able to easily review the port scripts when the are updated. This unfortunately precludes the use of patch files because they are sensible to many small non-semantic changes that can occur during a version update and that, all together, can hide to a human reviewer a single semantic change (think for example to the removal of a single null check hidden in a large patch-set addressing whitespace changes, #include
replacements and so on).
For this reason we are going to use dedicated easy to read scripts for every single transformations that the sources will require: for example during a review it is much easier to ensure that a remove-stdio-include.sed
script behaves correctly, than the same transformation spread in dozen of patch files (or in one patch targeting multiple files).
In the long term, when Harvey will be a mainstream OS supported like any other by the upstream developers, the vendor tool and its automatic checker will be still useful for updates: we will only need to remove the porting scripts from the build,json
.
One of my CLs went off the rails a bit and did this before getting reigned in, but it'll turn out better if discussed: how should util/build work if the BUILD script in the root is removed? (The implicit assumption here is that it'd be nice if it was removed.)
BUILD does some sequencing of util/build invocations, so that would have to be encoded in the json buildfiles somehow.
Should it understand barewords, like all
or utils
or libs
? Should it have a default filename?
I think cramming too much more functionality into it gets away from a simple tool that can be told how to invoke the toolchain.
That function was meant to be called with the queue locked. #mnt doesn't
lock it - nor does it or any other file have a mechanism to do so. So it's
always a bad idea to call qremove. It looks like #mnt wants qget().
Not sure if this is a problem in Plan 9 / Harvey or not.
kern/drivers/dev/mnt.c | 2 +-
kern/include/ns.h | 1 -
kern/src/ns/qio.c | 2 +-
3 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/kern/drivers/dev/mnt.c b/kern/drivers/dev/mnt.c
index b9036cc438bb..df3dd304b319 100644
--- a/kern/drivers/dev/mnt.c
+++ b/kern/drivers/dev/mnt.c
@@ -969,7 +969,7 @@ int mntrpcread(struct mnt *m, struct mntrpc *r)
l = &r->b;
*l = NULL;
do {
b = qremove(m->q);
b = qget(m->q);
/\* TODO: have better block helpers for this and the memmove below _/
b = linearizeblock(b);
if (hlen > 0) {
2.8.0.rc3.226.g39d4020
Compilation without errors.
Compilation errors with:
2016/03/23 21:01:08 "mksys -o sys.h '-mode=sys.h' $HARVEY/sys/src/sysconf.json " | /bin/sh
/bin/sh: 1: mksys: not found
I think that setting the HARVEY environment variable is the problem. (On Linux)
Since we need to focus on a working core system we have to postpone the porting and modernization of some valuable tools that we inherited from Plan 9.
To clean up (and to clearly define) our core system, without loosing the connection with our glorious past, we need to move such valuable tools to Harvey-OS/legends, where their portings will be addressed with a lower priority.
Each tool should be moved with related documentation and manual pages in a directory tree resembling its original location.
If you cat /bin/rc it seems like the console server always returns EOF on read from that point on. Or it's dying? I'm not sure.
We should:
const
modifiers where appropriate in function signaturesint
arguments with uint8_t
in memccpy, memchr and memset (and check if we have any performance loss)gmtime
and localtime
to get a Tm *tm
to fill instead of returning static data whose content is overwritten by each call (return value should then be void)getcallerpc
signature: the firstarg parameter is not necessary.abort
to produce useful messagesBefore the section "push the changes to your new remote" there needs to be a section "git remote set-url USER_NAME https://github.com/grd/harvey.git"
Lock(2) functions do not handle timeouts. We need them.
We can easily implement a lockt
function based on tsemacquire
, but for QLock
, RWLock
and Rendez
we need a way to interrupt rendezvous(2).
Alarms
might work, but being based on notes, they are not efficient enough.
Download page maybe should be called 'getting started' but should not refer to gerrithub.io
Gerrit doesn't suck.
Gerrit sucks.
Use Gerrit.
We need to get Go test working, as they are an excellent stressor for the OS
With a root file system imported over 9p
bind -a '#K' /dev
echo -n opstart > /dev/kprofctl
cat /bin/* > /dev/null
takes a page fault in the kernel
9p is a dog. It has some fundamental design limits which seem to cause performance problems. One of them is the requirement that all operations on a connection be serialized. Another is that it's always one connection.The statefulness of 9p, which at one point seemed sensible, is now an anchor dragging along behind it.
We need to investigate newer protocols to see if they are better. Now that we can select from multiple mount devices, we could mount those new protocols over a new mount device and see if performance improves.
We can also investigate improvements to 9p but, note, the problem is much, much harder than it first seems.
We should always be able to build with -Werror
We should organize libmach so that we can ignore unsupported architectures using $ARCH
.
Once no #ifdef
appear in the code, we should remove unsupported architectures (one per CL) referencing this issue, so that we can simply revert the commit when someone will want to restore them.
Alvaro has pointed out that there are .h files that we can remove, now that ape is a separate repo. We need the list.
Please, see in our Google group last state of this issue.
The standard way in to Plan 9 is drawterm. We're close.
The general process will be:
Grab and verify the tarball as normal, then for every file that would get written out to disk, instead checksum against the preexisting file on disk.
This should require one boolean flag (-check
).
Man pages references are outdated.
Name of the operating system should be harvey
All links are working
References to Plan 9
https://sevki.io/harvey/sys/man/1/acme references to sam, sam https://sevki.io/harvey/sys/man/1/sam doesn't exist
Outdated app links are still in the pages
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.