Git Product home page Git Product logo

goavatar's Introduction

Goavatar

Local component that interfaces with external devices and the Octopus website.

Overview

bin/              scripts for installing and testing
cmd/
    obf/          the Octopus Binary Format viewer        (see: obf --help)
    octopus/      the Octopus Connector websocket server  (see: octopus --help)
    printer/      print device streams in the console     (see: printer --help)  
datastruct/       data structures used in the application layer
device/           application layer generic device
drivers/          devices we currently support
    avatar/       the AvatarEEG
    mock_avatar/  a fake AvatarEEG for testing
    thinkgear/    devices with NeuroSky ThinkGear protocol
etc/              tools for testing
formats/          codecs and file recorders for OBF
socket/           the octopus socket and protocol
util/             generic utilities
var/              empty directory for testing files

Installation

Use Go 1.1 (or later). Make sure your $GOPATH is set. Make sure $GOPATH/bin is on your PATH. To get the repo:

$ go get -u -v github.com/jbrukh/goavatar/...

This installs the repo into $GOPATH/src/github.com/jbrukh/goavatar. If you are not finding dependencies, then try go-getting them:

$ go get <package>

To compile, test, and install everything:

$ bin/release.sh

To recompile certain builds you sometimes need to do:

$ go clean -i && bin/release.sh

Devices

Commands that take command-line parameters will usually take:

-device="avatar": one of {'avatar', 'mock_avatar', 'thinkgear'}
-mockChannels=2: the number of channels to mock in the mock device
-mockDevice=false: whether to use the mock device
-mockFile="etc/1fabece1-7a57-96ab-3de9-71da8446c52c": OBF file to play back in the mock device
-port="/dev/tty.AvatarEEG03009-SPPDev": the serial port for the device
-repo="var": directory where recordings are stored

The option --mockDevice is short for --device="mock_avatar". Usually the NeuroSky MindBand lives on port /dev/tty.BrainBand-DevB, so you would run like this:

$ octopus --device=thinkgear --port="/dev/tty.BrainBand-DevB"

To simulate multiple channels when using the MockAvatarEEG, do:

$ octopus --mockDevice --mockChannels=8

Protocol

//---------------------------------------------------------//
// Messages
//---------------------------------------------------------//

type (

    // Base type for messages.
    Message struct {
        Id          string `json:"id"`           // should be non-empty
        MessageType string `json:"message_type"` // will be one of {"info", connect", "record", "upload", "error"}
    }

    // Basic information about the server.
    InfoMessage struct {
        Id          string `json:"id"`           // should be non-empty
        MessageType string `json:"message_type"` // should be "info"
    }

    // ConnectMessage is used to connect to the device
    // and begin streaming. A ConnectResponseMessage is
    // sent to indicate success or failure, and data
    // immediately begins to flow on the data endpoint.
    ConnectMessage struct {
        Id          string `json:"id"`           // should be non-empty
        MessageType string `json:"message_type"` // should be "connect"
        Connect     bool   `json:"connect"`      // boolean to engage or disengage the device
        Pps         int    `json:"pps"`          // points per second, one of 250, 125, 83, ..., 250/k
        BatchSize   int    `json:"batch_size"`   // points to return per batch
    }

    // RecordMessage is used to trigger recording on
    // a device connection that is engaged. A RecordResponseMessage
    // is sent to indicate success (if recording has commenced) or
    // failure (if the device is off, or other errors).
    RecordMessage struct {
        Id          string `json:"id"`           // should be non-empty
        MessageType string `json:"message_type"` // should be "record"
        Record      bool   `json:"record"`       // start or stop recording
        Seconds     int    `json:"seconds"`      // number of seconds after which to cease recording
    }

    // UploadMessage is used to trigger upload of a
    // recorded resource available in the local repository.
    UploadMessage struct {
        Id          string `json:"id"`           // should be non-empty
        MessageType string `json:"message_type"` // should be "upload"
        Token       string `json:"token"`        // authentication token for upload
        ResourceId  string `json:"resource_id"`  // id of the resource to upload
        Endpoint    string `json:"endpoint"`     // domain-qualified endpoint to upload to
        Clear       bool   `json:"clear"`        // delete the file after upload?
    }

    // RepositoryMessage performs operations on the
    // device repository.
    RepositoryMessage struct {
        Id          string `json:"id"`           // should be non-empty
        MessageType string `json:"message_type"` // should be "repository"
        Operation   string `json:"operation"`    // one of {"list", "clear", "delete", "get"}
        ResourceId  string `json:"resource_id"`  // delete a specific file, in the case of "delete" or "get"
    }

    // Base type for response messages.
    Response struct {
        Id          string `json:"id"`           // echo of your correlation id
        MessageType string `json:"message_type"` // will be one of {"info", connect", "record", "upload", "error"}
        Success     bool   `json:"success"`      // whether or not the control message was successful
        Err         string `json:"err"`          // error text, if any
    }

    // ConnectResponseMessage is sent in response to a ConnectMessage.
    // The MessageType is set to "connect".
    ConnectResponse struct {
        Id          string `json:"id"`           // echo of your correlation id
        MessageType string `json:"message_type"` // will be "connect"
        Success     bool   `json:"success"`      // whether or not the control message was successful
        Err         string `json:"err"`          // error text, if any
        Status      string `json:"status"`       // device status, one of {"armed", "busy", "disconnected"}
    }

    // RecordResponseMessage is sent in response to a RecordMessage.
    // The MessageType is set to "record".
    RecordResponse struct {
        Id          string `json:"id"`           // echo of your correlation id
        MessageType string `json:"message_type"` // will be "record"
        Success     bool   `json:"success"`      // whether or not the control message was successful
        Err         string `json:"err"`          // error text, if any
        ResourceId  string `json:"resource_id"`  // id of the resource
        Seconds     int    `json:"seconds"`      // number of seconds recorder if this was a fixed-time recording
        SessionId   string `json:"session_id"`   // the connector session that made this recording (see InfoResponse.SessionId)
    }

    // UploadResponse is sent in response to an UploadMessage, providing
    // the URL of the uploaded resource.
    UploadResponse struct {
        Id          string `json:"id"`           // echo of your correlation id
        MessageType string `json:"message_type"` // will be "upload"
        Success     bool   `json:"success"`      // whether or not the control message was successful
        Err         string `json:"err"`          // error text, if any
    }

    // InfoResponse sends back information about the device and server.
    InfoResponse struct {
        Id          string `json:"id"`           // echo of your correlation id
        MessageType string `json:"message_type"` // will be "info"
        Success     bool   `json:"success"`      // whether or not the control message was successful
        Err         string `json:"err"`          // error text, if any
        Version     string `json:"version"`      // octopus server version
        DeviceName  string `json:"device_name"`  // device name
        SessionId   string `json:"session_id"`   // session id, lives for the life of control socket connection
    }

    // RepositoryResponse sends back messages about repository operations.
    RepositoryResponse struct {
        Id            string          `json:"id"`             // echo of your correlation id
        MessageType   string          `json:"message_type"`   // will be "repository"
        Success       bool            `json:"success"`        // whether or not the control message was successful
        Err           string          `json:"err"`            // error text, if any
        ResourceInfos []*ResourceInfo `json:"resource_infos"` // list of files and infos
    }

    // Resource information from the repo.
    ResourceInfo struct {
        Id           string `json:"id"` // this is the resourceId
        File         string `json:"file"`
        SizeBytes    int64  `json:"size_bytes"`
        LastModified int64  `json:"last_modified"`
    }

    // DataMessage returns datapoints from the device across
    // the channels. These data points represent incremental data
    // that has not been seen before. The data messages come at a
    // frequency specified in the initial control messages.
    DataMessage struct {
        Data      [][]float64 `json:"data"`       // the data for each channel, only first n relevant, n == # of channels
        Ints      [][]int64   `json:"ints"`       // the data for each channel, as integers
        LatencyMs float64     `json:"latency_ms"` // the running latency
        //Timestamp int64      `json:"timestamp"` // timestamp corresponding to this data sample

    }
)

OBF Viewer

Use the obf command to view OBF files.

Usage of obf:
  -csv=false: output strict CSV
  -humanTime=false: format timestamps
  -plot=false: ouput the series on a gplot graph
  -seq=false: read sequential data, if available

For example:

$ time obf var/12fb51e907ea112a | head -40
# Octopus Binary Format.
#
# Copyright (c) 2013. Jake Brukhman/Octopus.
# All rights reserved.
#
# HEADER ----------------------------------
# DataType:       1
# FormatVersion:  1
# StorageMode:    1
# Channels:       2
# Samples:        1520
# SampleRate:     250
# ------------------------------------------
timestamp,channel1,channel2
1345796045054931640,0.50088380277156829834,0.50130996108055114746
1345796045058931640,0.50079300999641418457,0.50126963853836059570
1345796045062931640,0.50051495432853698730,0.50136803090572357178
1345796045066931640,0.50027900934219360352,0.50124102830886840820
1345796045070931640,0.50031083822250366211,0.50137455761432647705
1345796045074931640,0.50034177303314208984,0.50138631463050842285
1345796045078931640,0.50037132203578948975,0.50136722624301910400
1345796045082931640,0.50039720535278320312,0.50129100680351257324
1345796045086931640,0.50042653083801269531,0.50134509801864624023
1345796045090931640,0.50045697391033172607,0.50128161907196044922
1345796045094931640,0.50048710405826568604,0.50140835344791412354
1345796045098931640,0.50051616132259368896,0.50131402909755706787
1345796045102931640,0.50054503977298736572,0.50132882595062255859
1345796045106931640,0.50057396292686462402,0.50134214758872985840
1345796045110931640,0.50060141086578369141,0.50126892328262329102
1345796045114931640,0.50062935054302215576,0.50138886272907257080
1345796045118896484,0.50065769255161285400,0.50123070180416107178
1345796045122896484,0.50068885087966918945,0.50134089589118957520
1345796045126896484,0.50071911513805389404,0.50133928656578063965
1345796045130896484,0.50074808299541473389,0.50131858885288238525
1345796045134896484,0.50077910721302032471,0.50128304958343505859
1345796045138896484,0.50081008672714233398,0.50137670338153839111
1345796045142896484,0.50084093213081359863,0.50131501257419586182
1345796045146896484,0.50086618959903717041,0.50137008726596832275
1345796045150896484,0.50089269876480102539,0.50136709213256835938
1345796045154896484,0.50092291831970214844,0.50139266252517700195

real    0m0.018s
user    0m0.013s
sys 0m0.005s

License

Copyright (c) 2013. Jake Brukhman/Octopus.

I am hereby releasing this software under the MIT License.

goavatar's People

Contributors

jbrukh avatar jonnii avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

goavatar's Issues

Connector binary stuff

I took a look at the code and I understand what's going on now.

In session.go there's a method called ProcessRepositoryMessage. You build a repository response and defer send it to the client, if the message is a get call then it sends the file to the client and after sends the repository response (because we always have a response to every message).

In the client we're doing this:

connector.send('repository', { operation: 'get', resource_id: resourceId }).then (data) =>
      console.log data

The data the connector receives is of type Blob but the callback is never called with that data for a few reasons:

  1. We don't yet support binary data on the client (lols)
  2. The client tries to parse the data as json (lols)
  3. The client doesn't know which connector callback to call with this data, because it doesn't have a correlation id.

I can fix 1 and 2, but the binary response from the connector needs a correlation id, I suggest that when we send a file from the connector the first byte is the correlation id (I know the body is a type of Blob or a string).

Once this is done the next problem is that the connector will also send a json response to the client for the original request, this will cause problems.

Short version is do the following:

  • On get send the file with the first byte(s) being the correlation id of the original message (the id).
  • Don't send the json response that's currently being deferred.

I wanted to try to fix this but I don't know enough go yet. I'm going to make the client changes though.

LMK

Bug: pps/batchSize in socket is not thread-safe.

Ok, it is currently thread-safe, but fragile. Data socket will block on the kickoff channel before reading the values which were guaranteed to be written beforehand. Think of alternative solutions.

Make sure that the connector can only access files in the repository

It's likely that anyone who looks into how we've done local recordings may have questions about security. Most of these will be based upon the fact that I might be able to read any file on your hard drive. What happens if I send a request like:

connector.send('repository', { operation: 'get', resource_id: '../../../../another_file' }).then (data) =>
Let's just double check that the repository can't access anything outside of its sandbox.

Split cloud/local recordings into separate directories

When I do a cloud recording it also appears in the local tab. We should add a flag to "record" to indicate if the recording is local. If it's local, then put it into /var/local or /var/cloud.

When I upload a local recording move it from /var/local to /var/cloud.

repository commands should also have this flag, so we can manage the /var/cloud directory as well as the /var/local directory.

Meta data about local recordings

There will come a time when someone is going to ask us to give a local recording a name. We might need a local sql server like sqlite or something. Maybe worth investigating.

Crash when using mockChannels=8

Set 8 mock channels, tried to record, got the following:

➜ goavatar git:(master) octopus --mockDevice --mockChannels=8
Device: MockAvatarEEG
Control: http://localhost:8000/control
Data: http://localhost:8000/device
socket.go:83: SOCKET: RECEIVED {"id":"1368748028560","message_type":"info"}
socket.go:161: SOCKET RESPONSE: &{Id:1368748028560 MessageType:info Success:true Err: Version:0.1 DeviceName:MockAvatarEEG}
socket.go:83: SOCKET: RECEIVED {"connect":true,"pps":50,"batch_size":1,"id":"1368748036789","message_type":"connect"}
socket.go:161: SOCKET RESPONSE: &{Id:1368748036789 MessageType:connect Success:true Err: Status:armed}
device.go:201: MockAvatarEEG: CONNECT
mock.go:41: loaded mock data with 189 data frames
socket.go:161: SOCKET RESPONSE: &{Id: MessageType:connect Success:true Err: Status:streaming}
socket.go:380: DEVICE: STREAMING ON
socket.go:83: SOCKET: RECEIVED {"record":true,"id":"1368748044434","message_type":"record"}
device.go:295: MockAvatarEEG: RECORD
socket.go:161: SOCKET RESPONSE: &{Id:1368748044434 MessageType:record Success:true Err: ResourceId:}
panic: bad parameters

goroutine 6 [running]:
github.com/jbrukh/goavatar.NewBlockBuffer(0x8, 0x1, 0x0, 0x331900)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/block_buffer.go:35 +0x10f
github.com/jbrukh/goavatar/devices/mock_avatar.(_MockDevice).transformBuffer(0xf840001a00, 0xf8400bd5d0, 0xf8400bd5d0, 0xf8400fb7b0, 0xf8400bd5a0, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/devices/mock_avatar/mock_device.go:109 +0x8d
github.com/jbrukh/goavatar/devices/mock_avatar.(_MockDevice).getFrame(0xf840001a00, 0xf800000000, 0xf8400f7970, 0xf8400f7970, 0x0, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/devices/mock_avatar/mock_device.go:96 +0xfb
github.com/jbrukh/goavatar/devices/mock_avatar.(_MockDevice).Stream(0xf840001a00, 0xf840123840, 0x0, 0x0, 0x0, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/devices/mock_avatar/mock_device.go:60 +0x9a
github.com/jbrukh/goavatar._func_001(0xf840092818, 0x0)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/device.go:214 +0x6b
created by github.com/jbrukh/goavatar.(_BaseDevice).Connect
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/device.go:225 +0x340

goroutine 1 [chan receive]:
net.(_pollServer).WaitRead(0xf840068b80, 0xf8400b12d0, 0xf840066f30, 0x23, 0x1, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/fd.go:268 +0x73
net.(_netFD).accept(0xf8400b12d0, 0xe6e5f, 0x0, 0xf8400665a0, 0xf840092040, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/fd.go:622 +0x20d
net.(_TCPListener).AcceptTCP(0xf8400922b0, 0xf840078c00, 0x0, 0x0, 0x8, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/tcpsock_posix.go:320 +0x71
net.(_TCPListener).Accept(0xf8400922b0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/tcpsock_posix.go:330 +0x49
net/http.(_Server).Serve(0xf840068ac0, 0xf840068c40, 0xf8400922b0, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:1029 +0x88
net/http.(_Server).ListenAndServe(0xf840068ac0, 0xf840068ac0, 0xf840068a40, 0x2448)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:1019 +0xb6
net/http.ListenAndServe(0xf840092250, 0x7665642f00000005, 0x0, 0x0, 0xf840068a40, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:1091 +0x69
main.main()
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/cmd/octopus/octopus.go:55 +0x448

goroutine 2 [syscall]:
created by runtime.main
/usr/local/Cellar/go/1.0.3/src/pkg/runtime/proc.c:221

goroutine 3 [syscall]:
syscall.Syscall6()
/usr/local/Cellar/go/1.0.3/src/pkg/syscall/asm_darwin_amd64.s:38 +0x5
syscall.kevent(0x6, 0x0, 0x0, 0xf8400a5188, 0xa, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/syscall/zsyscall_darwin_amd64.go:199 +0x88
syscall.Kevent(0xf800000006, 0x0, 0x0, 0xf8400a5188, 0xa0000000a, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/syscall/syscall_bsd.go:546 +0xa4
net.(_pollster).WaitFD(0xf8400a5180, 0xf840068b80, 0x0, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/fd_darwin.go:96 +0x185
net.(_pollServer).Run(0xf840068b80, 0x0)
/usr/local/Cellar/go/1.0.3/src/pkg/net/fd.go:236 +0xe4
created by net.newPollServer
/usr/local/Cellar/go/1.0.3/src/pkg/net/newpollserver.go:35 +0x382

goroutine 4 [chan receive]:
net.(_pollServer).WaitRead(0xf840068b80, 0xf8400b1360, 0xf840066f30, 0x23, 0x1, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/fd.go:268 +0x73
net.(_netFD).Read(0xf8400b1360, 0xf8400c7000, 0x100000001000, 0x1000000ffffffff, 0xf8400665a0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/fd.go:428 +0x1ec
net.(_TCPConn).Read(0xf840092318, 0xf8400c7000, 0x100000001000, 0x2381000, 0x0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/tcpsock_posix.go:87 +0xce
io.(_LimitedReader).Read(0xf84006b920, 0xf8400c7000, 0x100000001000, 0xfbe00000000, 0x0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/io/io.go:406 +0xc1
bufio.(_Reader).fill(0xf840068dc0, 0x23c2b08)
/usr/local/Cellar/go/1.0.3/src/pkg/bufio/bufio.go:77 +0xf0
bufio.(_Reader).ReadByte(0xf840068dc0, 0x1cf100, 0x0, 0x0, 0xf8400d6640, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/bufio/bufio.go:166 +0x83
code.google.com/p/go.net/websocket.hybiFrameReaderFactory.NewFrameReader(0xf840068dc0, 0xf8400d60a0, 0xf8400d6640, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/code.google.com/p/go.net/websocket/hybi.go:115 +0xd0
code.google.com/p/go.net/websocket.Codec.Receive(0x974d0, 0x976a5, 0xf8400da000, 0x133aa0, 0xf8401fa050, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/code.google.com/p/go.net/websocket/websocket.go:315 +0x16e
github.com/jbrukh/goavatar/socket.(_SocketController).Receive(0xf84006bfa0, 0xf84007d800, 0x6000000003c, 0xf8401e2e90, 0xd, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/socket/socket.go:125 +0xba
github.com/jbrukh/goavatar/socket._func_001(0xf840078720, 0x9642e, 0xf8400da000, 0xf840092318, 0xf840078790, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/socket/socket.go:75 +0xda
code.google.com/p/go.net/websocket.Handler.ServeHTTP(0xf840068a00, 0xf84006ca00, 0xf84006c9c0, 0xf8400d2000, 0xf800000000, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/code.google.com/p/go.net/websocket/server.go:76 +0x210
net/http.(_ServeMux).ServeHTTP(0xf84006b580, 0xf84006ca00, 0xf84006c9c0, 0xf8400d2000, 0xf84006c9c0, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:941 +0xb9
net/http.(_conn).serve(0xf840067840, 0x0)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:669 +0x621
created by net/http.(_Server).Serve
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:1057 +0x430

goroutine 5 [runnable]:
github.com/jbrukh/goavatar/socket.streamLoop(0xfa00000008, 0xf84013d000, 0xf8400bd500, 0xf8400da200, 0x23dfc01, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/socket/socket.go:438 +0x113
github.com/jbrukh/goavatar/socket.stream(0xf8400da200, 0xf8400001c0, 0xf8400689c0, 0xf8400d6200, 0x1f0430, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/socket/socket.go:407 +0x194
github.com/jbrukh/goavatar/socket._func_002(0xf840078730, 0xf840092258, 0x9642e, 0xf8400da200, 0xf8400927e0, ...)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/socket/socket.go:364 +0x2c3
code.google.com/p/go.net/websocket.Handler.ServeHTTP(0xf840068a40, 0xf84006ca00, 0xf840065140, 0xf8400a4000, 0xf800000000, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/code.google.com/p/go.net/websocket/server.go:76 +0x210
net/http.(_ServeMux).ServeHTTP(0xf84006b580, 0xf84006ca00, 0xf840065140, 0xf8400a4000, 0xf840065140, ...)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:941 +0xb9
net/http.(_conn).serve(0xf8400df360, 0x0)
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:669 +0x621
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.0.3/src/pkg/net/http/server.go:1057 +0x430

goroutine 7 [timer goroutine (idle)]:
created by addtimer
/usr/local/Cellar/go/1.0.3/src/pkg/runtime/ztime_amd64.c:72

goroutine 8 [chan receive]:
github.com/jbrukh/goavatar/formats.worker(0xf840067ea0, 0x0)
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/formats/obf_recorder.go:59 +0x8b
created by github.com/jbrukh/goavatar/formats.(*OBFRecorder).Start
/Users/jonnii/gocode/src/github.com/jbrukh/goavatar/formats/obf_recorder.go:45 +0x5a

Send length of recording as part of recording response

Right now I have a background job that will process obf files and calculate how long they are, in prod this involves paying for a worker, we should calculate this on the client.

Right now I think it's sending Seconds:n, can we change that to be milliseconds?

Timed Recording not sending resource_id

This might be a problem on my end, but I'm not sure. The logs are below. It looks like I'm getting multiple record messages when starting a timed record, with and EOF err on the second one, and then no resourceId to upload.

Beginning timed recording of 3 seconds recordings_new_controller.js:37
start recording recordings_new_controller.js:41
Sending connector message: record connector.js:66
Connector response connector.js:83
Object {id: "1369690505087", message_type: "record", success: true, err: "", resource_id: ""…}
 connector.js:84
Connector response connector.js:83
Object {id: "1369690505087", message_type: "record", success: true, err: "EOF", resource_id: ""…}
 connector.js:84
timed recording finished recordings_new_controller.js:45
End Record recordings_new_route.js:33
Sending connector message: record connector.js:66
Connector response connector.js:83
Object {id: "1369690508120", message_type: "record", success: false, err: "", resource_id: ""…}
 connector.js:84
end record received recordings_new_route.js:39
Commiting transaction recordings_new_route.js:60
created current model recordings_new_route.js:44
Sending connector message: upload connector.js:66
Connector response connector.js:83
Object {id: "1369690508260", message_type: "upload", success: false, err: "read var: is a directory"} connector.js:84
stopping experiment recordings_new_controller.js:12
stopping streaming web_socket_data_adapter.js:59
Sending connector message: connect connector.js:66
Connector response connector.js:83
Object {id: "1369690508271", message_type: "connect", success: true, err: "", status: "disarmed"} connector.js:84
data socket closed web_socket_data_adapter.js:49

Service logs

device.go:164: MockAvatarEEG: CONNECT
mock.go:41: loaded mock data with 189 data frames
device.go:194: MockAvatarEEG: DEVICE INFO &{Channels:4 SampleRate:250}
handler.go:157: Octopus Socket: RESPONDED &{Id: MessageType:connect Success:true Err: Status:streaming}
data.go:40: MockAvatarEEG: STREAMING ON
handler.go:109: Octopus Socket: RECEIVED {"record":true,"seconds":3,"id":"1369690505087","message_type":"record"}
session.go:188: FIXED TIME RECORDING: 3 seconds, 750 points
handler.go:157: Octopus Socket: RESPONDED &{Id:1369690505087 MessageType:record Success:true Err: ResourceId: Seconds:0}
obf_recorder.go:82: OBFRecorder: opening file for writing: var/27fed955-9a79-b943-e9cc-5ff981a1868f
obf_recorder.go:90: OBFRecorder: closing the file: var/27fed955-9a79-b943-e9cc-5ff981a1868f
session.go:199: error during fixed-time recording: EOF
handler.go:157: Octopus Socket: RESPONDED &{Id:1369690505087 MessageType:record Success:true Err:EOF ResourceId: Seconds:3}
handler.go:109: Octopus Socket: RECEIVED {"record":false,"id":"1369690508120","message_type":"record"}
handler.go:157: Octopus Socket: RESPONDED &{Id:1369690508120 MessageType:record Success:false Err: ResourceId: Seconds:0}
handler.go:109: Octopus Socket: RECEIVED {"token":"sPDp2A2kkobxnNi2gYBA","resource_id":"","endpoint":"http://localhost:3000/api/recordings/14/results","id":"1369690508260","message_type":"upload"}
upload.go:20: uploading file var to endpoint: http://localhost:3000/api/recordings/14/results
handler.go:157: Octopus Socket: RESPONDED &{Id:1369690508260 MessageType:upload Success:false Err:read var: is a directory}
handler.go:109: Octopus Socket: RECEIVED {"connect":false,"id":"1369690508271","message_type":"connect"}
device.go:214: MockAvatarEEG: DISCONNECT
handler.go:157: Octopus Socket: RESPONDED &{Id:1369690508271 MessageType:connect Success:true Err: Status:disarmed}
data.go:47: MockAvatarEEG: STREAMING OFF

Improve reliability of parsing logic.

Improvement will happen thus: for each frame, peek until the frame size; check the frame size against max frame size; if it is acceptable, then peek the entire frame at once and check CRC. If CRC fails, just continue until next sync. If CRC succeeds, then parse the frame.

At this point we have the option to check the frame count, and if frames have been dropped, then we can return empty frames.

Design binary format for storing raw device data.

Proposed format:

[DataType] -- 1 byte -- Describes what type of data is stored in this file. 0x01 = Raw device data. 

[FormatVersion] -- 1 byte -- Version of the ensuing format, allowing for revisions of format. 0x01 = v1

[Storage Mode] -- 1 byte -- 0x00 = parallel, 0x01 = sequential. 

[Channels] -- 1 byte -- The number of channels, between 0 and 255. 

[Samples] -- int32 -- The number of samples (~50 days of recording max). 

[Values...] -- float64 * channels * samples -- values either in parallel or sequential format. 

[Timestamps...] -- (int64+int64)*channels -- TODO... how are these stored? relative or actual time?

Design binary format for storing raw device data.

Proposed format:

[DataType] -- 1 byte -- Describes what type of data is stored in this file. 0x01 = Raw device data. 

[FormatVersion] -- 1 byte -- Version of the ensuing format, allowing for revisions of format. 0x01 = v1

[Storage Mode] -- 1 byte -- 0x00 = parallel, 0x01 = sequential. 

[Channels] -- 1 byte -- The number of channels, between 0 and 255. 

[Samples] -- int32 -- The number of samples (~50 days of recording max). 

[Values...] -- float64 * channels * samples -- values either in parallel or sequential format. 

[Timestamps...] -- int64*channels -- timestamps corresponding to the values.

Windows build

Do we need to build it on a windows machine? I have a VM we could use.

Add a unique device instance id for verifying uploads

On connect give me a unique id, which should change every time connector is restarted. Can just be a timestamp of when the connector was started or a guid. When uploading send this unique id, so it can be verified against the recording.

On upload also include:

  • device name
  • connector version
  • any other information that might be useful for diagnosing recording problems

Change OBF format timestamps

Instead of unix timestamps can we please have it as the milliseconds from the start of the recording. This means we can reduce the size of the timestamps field from 64bit to 32bit.

Artificial Latency

Is there any way we could add a flag which adds some artificial latency to the collector so it's easier to test repository latency in the UI.

Long-term restructuring of OBF format and implementation.

  • change the file name to UUIDs
  • include a SHA that checks integrity of the file
  • perform conversion of formats/SHA hashing at record STOP for optimized performance
  • P mode and S mode will be stored in the same file, with mode "Combined".
  • finish OBFWriter interface
  • provide support for subsequencing
  • look into gzipping

Deprecate DataFrame.

All streaming will be done with BlockBuffers; sampleRate info shall be handed off separately.

Upload message broken

Need to specify local. If local, then move the file from /var/local to /var/cloud.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.