imaman / bigband Goto Github PK

currently, when a lambda instrument uses a dynamodb instrument it is granted full access to the respective dynamodb table. We can distinguish between various access needs (an enum passed to the dynamodb instrument? a wrapper-instrument?) and have just the minimal set of permissions granted.

the bigband file specifies multiple names. in many cases these names translate (on their own, or by composing several names together) into names of cloudformation entities/AWS resources. We need to apply the limits set by AWS to have the Bigband system fail as early as possible (As opposed to having the failure reported by cloudformation).

Aspects to consider:

• name length limit
• valid characters

as opposed to injecting names/ARNs and having the lambda code create the respective client on its own.

currently a handler file must define a file-level function. if the user wants to use a controller she must create it/cache it in the function

https://docs.aws.amazon.com/cdk/latest/guide/serverless_example.html

runs bigband-example (or a similar script) to exercise the Bigband CLI end-to-end

• should pick a random bigband name (GUID, probably)
• possible steps: create a new bigbnad with a single lambda, invoke it, make some arch. changes, push it, invoke it, add a dep. ship, invoke, make some code changes ship, cleanup, verify Cloudformation stack is gone.
• assertions: on each push parse the teleporting message and check number of bytes uploaded. on each invocation verify that output

by running this command a new lambda instrument will be created: the entry point file will be created with some default content + it will print a snippet to place into the bigband file.

per-section: enables multi-region bigbands (with separate sections residing in possibly separate regions)
per-isolation scope: prevent costs due to cross-region access to the s3 bucket

In https://readthedocs.org or a similar service.

Naturally, this is blocked on #43

we have table instrument, we need also an index instrument.

Physical names have a strict size limit which of 64 character[1]. Given that the physical name encode the bigband name, the section name, an optional path, and an instrument name it is easy to hit this limit when trying to give meaningful names.

possible non-mutually-exclusive suggestions:
(1) use camelCasing (may require enforcing all-lower case names) when concatenating the physical names to save space (as "-" separators are going to be omitted that way)
(2) drop well known suffixes such as "-service" "-section". may not work as one can have the same prefix for two different instruments, such as "user-storage" (a dynamodb table) and "user-service" (a lambda function). but maybe there is some way to make it work
(3) strict length restriction on section or bigband name (not sure I like this idea)

in parallel to these directions we should still provide human-friendly naming to users of the CLI. That is, even if the physical name may be an ugly camel-case name ("myBigbandMySectionMyPackageMyFunction") we can still use dash-/comma- separated names in the CLI's UX ("my-bigband.my-section.my-package.myFunction")

[1] https://docs.aws.amazon.com/lambda/latest/dg/API_CreateFunction.html#SSS-CreateFunction-request-FunctionName

There is a --incremental flag (introduced in TS 3.4):

Enable incremental compilation by reading/writing information from prior compilations to a file on disk. This file is controlled by the --tsBuildInfoFile flag.

https://www.typescriptlang.org/docs/handbook/compiler-options.html

This may affect #38

Building on top of #23 this instrument will allow bigband users to define a web-site to be served off-of S3. The JS bundle sent to the browser (when accessing this site) should include some mechanism for accessing other instruments (specifically, lambda functions/API gateway).

There is considerable design work here. No need to support all JS frameworks from day one.

bigband ls p1   # Show all instruments in the p1 package

bigband ls p1-p2  # Show all instruments in the p1-p2 package

bigband ls p1-p2-p3-myLambda  # shows resources available for bigband cat. Sepcfically: log file, info

bigband cat p1-p2-p3-myLambda/log
bigband cat p1-p2-p3-myLambda/info

bigband exec p1-p2-p3-myLambda '{"a": 1, "b": 2}'

Alternatively, perhaps the command "ls", "cat", "exec" is an extension to the last component in the path (with the default being "ls"), so that one can do:

bigband p1-p2-p3
bigband p1-p2-p3-myLambda 
bigband p1-p2-p3-myLambda.exec '{"a": 1, "b": 2}'
bigband p1-p2-p3-myLambda.log
bigband p1-p2-p3-myLambda.info

this results in even shorter command lines but it may be confusing to newcomers.

CF will take care of deleting resources (spec. instruments) when they are removed from the section. However, if a section is removed from the bigband, the bigband tool does not delete the corresponding CF stack.

For regions, we may want to delete the bucket.

One approach to do that is to provide a special section value which will indicate to the Bigband tool that the section is intended to be deleted.

show what are the differences between the local configuration to prod.

Currently, the user needs to have an s3 bucket created prior to using bigband. We can create one for the user. The big win here is that the user will not need to have the AWS CLI installed on her machine.

Can be done either automagically or via an explicit command line flag. I guess automagically provides a better UX.

If automagically we need to figure out the right region to create the bucket in.

for storage use either the actual resources from AWS or use a locally-running fakes.

Design work ahead.

Don't allow to do anything if there is a cycle

To allow auth-ing via a public identity provider

(a good starting point: https://stackoverflow.com/questions/51089397/restrict-login-to-enterprise-google-domain-for-aws-federated-identity-pool/52358653#52358653)

will create a fully functioning bigband (probably the user needs to provide a few details first). main purpose: ease the onboarding of newcomers.

Bigband will take an options argument (similar to the one passed to IsolationScope.create())

Consider: support passing a similar options object into Section to allow it override the ones from Bigband. May complicate things considerably. OTOH, may provide higher flexibility needed for future use-cases.

Error Error: (instrument: "bigband-system-teleport", rootDir: "/home/imaman/code/imaman/bigband/packages/cli", npmPackageDir: "/home/imaman/code/imaman/bigband/packages/example") The given instrument ("bigband-system-teleport") is not a member of the "prod-major" section
at SectionSpecModel.getWiringsOf (/home/imaman/code/imaman/bigband/packages/cli/src/SectionSpecModel.ts:16:19)
at /home/imaman/code/imaman/bigband/packages/cli/src/BigbandFileRunner.ts:300:15
at Generator.next ()
at /home/imaman/code/imaman/bigband/packages/cli/lib/BigbandFileRunner.js:7:71
at new Promise ()
at __awaiter (/home/imaman/code/imaman/bigband/packages/cli/lib/BigbandFileRunner.js:3:12)
at compileInstrument (/home/imaman/code/imaman/bigband/packages/cli/lib/BigbandFileRunner.js:258:12)
at /home/imaman/code/imaman/bigband/packages/cli/src/BigbandFileRunner.ts:276:34
at Generator.next ()
at /home/imaman/code/imaman/bigband/packages/cli/lib/BigbandFileRunner.js:7:71
Shipping section "prod-major" to eu-west-2
Compiling chronology-importantDates
Compiling geography-placeFinder
Compiling geography-analyzer
Compiling geography-healthChecker
Compiling bigband-system-teleport
Exiting: (instrument: "bigband-system-teleport", rootDir: "/home/imaman/code/imaman/bigband/packages/cli", npmPackageDir: "/home/imaman/code/imaman/bigband/packages/example") The given instrument ("bigband-system-teleport") is not a member of the "prod-major" section
No code changes in geography-healthChecker
No code changes in geography-analyzer
No code changes in chronology-importantDates
Teleported 0.000MB for geography-placeFinder

will show the name of the underlying cloudformation stack and similar details. it is possible that UX wise this could be part of the info resource discussed in #33

to be placed in S3 such that it is made available to other instruments (lambda comes to mind) at runtime.

This tasks requires figuring out the right API to expose to the bigband file.

auto-deletion of lambda functions is fine because you can easily rollback by means of source-control-reverting. However, for state-persisting instruments (dynamodb, s3) this is not the case. once auto-deleted was carried out, we cannot restore the data, and it is effectively lost.

we need to protect user from accidental deletion of state-persisting instruments. Two directions that come to mind:

• a special command line flag to signal the user's explicit will to delete such instruments (probably the flag should explicitly list the names of the instruments to be deleted)
• a special limbo zombie instrument to replace the actual instrument. when such a bigband is shipped deletion will occur. Otherwise, removal of a (non-zombie) state-persisting instrument will not result in an actual deletion.