There's one issue in Sentry I'm not sure why

https://sentry.io/organizations/dapperlabs/issues/3632555312/?project=6398442&query=is%3Aunresolved

Please investigate. One of the deliverables is to write a test that intentionally executes a transaction that fails and another one that succeeds and tests how the state recreation functions in such cases.

This case in the issue was somehow triggered due to the transaction succeeding when it first was executed but failed on state recreation, which shouldn't happen.

There's an issue with the emulator being nil. I believe this must be coming from concurrent management of resources, but it needs to be further investigated. This issue has only occurred on staging. There's no known reproducton.

Describe the feature few words

Create an API endpoint to duplicate a project.

How to impleneted

• Add a mutation to the GraphQL schema to duplicate a project based on the projectID.
• Create resolver for the mutation; get project with projectID, generate new projectID, make a copy of all files for the project (transaction_templates, script_templates, and contract_templates), create new project with copy of the files and return it.

What will be the end result of this change?

The end result will be to enable the front end to duplicate projects by copying the exact state into a new project.

Which components will this change affect?

This change will only effect the API through adding functionality.

Description

We would like to enable different number of accounts per projects to allow contracts, which require big number of import dependencies - TopShot for example.
But we don't want to simply increase number of accounts available, cause it will bloat the sidebar - especially on smaller screens.
When solved, this issue shall enable onflow/flow-playground#133 on front-end.

Solution

We will create additional field on the Project, which will hold current number of active accounts.

Possible Issues

One scenario, which can lead to some interaction issues would be if user would enable new accounts, then deploy contract to one of them and then reduce number back to initial number. If there will be imports from that address in other contracts, transactions or scripts, language client will not report any issues 🤷‍♂️

Describe the feature few words

Establish alerting for the /ping endpoint so we get notified if it's not working. Also refactor ping so it includes a database query to ensure the database connection is working.

What will be the end result of this change?

Improving health check for the playground.

Is there any previous work that needs to be completed before making this change?

N/A

Which components will this change affect?

• /ping endpoint

Other remarks

• Check with SRE on how we can get notified if /ping goes down automatically.

Improve the gen3 database migrator to migrate all projects. Also improve the performance.
Re-run the migration.

Describe the feature few words

Change the implementation of contracts, transaction templates, and script templates to be a file.

What will be the end result of this change?

The end result of this change will enable the front end and backend to treat all user code as a file rather than separate entities. The result of this change will be to have new models and graphql schema defined for the following:

• Refactor accounts to not include code
• Refactor contracts, script templates, and transaction templates into files; move deployed code to a model that will hold contract code
• The project can be refactored to not include accounts at all, as they are always sequential that can just become a counter on the project model

API will need to be added to include:

• Create a new file
• Update file name
• Delete file

Is there any previous work that needs to be completed before making this change?

No previous work is required.

Which components will this change affect?

This change will effect all API components, including the API itself as all functions will need to be re-implemented with the new models.
This change will effect the following components:

• Contract model
• Transaction template model
• Script template model
• GraphQL Schema

API Schema changes:

• Project now contains contractTemplates and numberOfAccounts, and no longer contains a list of accounts

The following types were added to support contracts:

• ContractTemplate, ContractDeployment

The following input types were added to support contracts:

• NewProjectContractTemplate, NewContractTemplate, UpdateContractTemplate , NewContractDeployment

The following API mutations were added for contracts:

• createContractTemplate(input: NewContractTemplate!): ContractTemplate!
• updateContractTemplate(input: UpdateContractTemplate!): ContractTemplate!
• deleteContractTemplate(id: UUID!, projectId: UUID!): UUID!
• createContractDeployment(input: NewContractDeployment!): ContractDeployment!

The following API mutations were removed:

• updateAccount(input: UpdateAccount!): Account!

The following API query was removed:

• account(id: UUID!, projectId: UUID!): Account!

The following API query was added:

• contractTemplate(id: UUID!, projectId: UUID!): ContractTemplate!

You can now query for deployed contracts and sort them into their addresses on the front end (instead of querying for an account which contains a list of deployed contracts)

Describe the feature few words

Create an API migrator to migrate projects from stable playground to v2.

What will be the end result of this change?

The end result is that users can access their old projects in the new playground.

Is there any previous work that needs to be completed before making this change?

Playground v2 file implementation must be completed first.

Which components will this change affect?

• Migrators

Other remarks

Describe the feature few words

Allow multiple contracts to be deployed to an account

What will be the end result of this change?

Allowing multiple contracts on same account, and if the same contract on same account is deployed, we should remove the contract and deploy it again without reseting the state

Is there any previous work that needs to be completed before making this change?

File refactor

Which components will this change effect?

• Contract Deployments chain

We need to migrate all data from the current datastore to the new SQL implementation.

• Change LRU cache implementation to a thread safe version: https://github.com/hashicorp/golang-lru/
• Add test cases to cache_test

There's an issue with cache invalidation. Current logic invalidates the cache based on the block height and the number of executions in the database. If a number of executions is lower in the database than in the cached emulator block height then it obviously needs to be reset. The problem is that there's an edge case where other executions are done after reset so the height matches.

There's an error reported on Sentry for overwriting the existing contract on the network. The problem is the existing contract shouldn't be overwritten, but there shouldn't be a way for that to happen since updates should be guarded by checking if the account has existing contracts and if it does it first resets the state.

See details https://sentry.io/organizations/dapperlabs/issues/3605855069/?project=6398442&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D&sort=inbox

A feature has been requested to have an endpoint where you can pass in a script and a blank text editor page will open with the script formatted with syntax highlighting (no other functionality needs to be present).

Cache updates seem to have an issue trying to update the cache. The error is seen here https://sentry.io/organizations/dapperlabs/issues/3623728662/?project=6398442&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D&sort=inbox

Try finding a solution to prevent this from happening.

Describe the bug
The script below causes the following error to be returned form the API:

Error: GraphQL error: failed to insert script execution record: datastore: string property too long to index for a Property with Name "Value"

To Reproduce
Attempt to execute a script of roughly the same number of characters.

Expected behavior
The script is executed properly. No errors.

Additional context
The script:

import NonFungibleToken from {{0xNonFungibleToken}}
import Basketballs from {{0xBasketballs}}
import BasketballsMarket from {{0xBasketballsMarket}}
pub struct MarketplaceSaleOfferView {
    pub let saleItemID: UInt64
    pub let salePrice: UFix64
    pub let saleCompleted: Bool
    pub let editionID: UInt32
    pub let serialNumber: UInt64
    pub let name: String
    pub let description: String
    pub let imageURL: String
    pub let nftCount: UInt64
    init(
      saleItemID: UInt64,
      salePrice: UFix64,
      saleCompleted: Bool,
      editionID: UInt32,
      serialNumber: UInt64,
      name: String,
      description: String,
      imageURL: String,
      nftCount: UInt64
    ) {
        self.saleItemID = saleItemID
        self.salePrice = salePrice
        self.saleCompleted = saleCompleted
        self.editionID = editionID
        self.serialNumber = serialNumber
        self.name = name
        self.description = description
        self.imageURL = imageURL
        self.nftCount = nftCount
    }
}
pub fun main(address: Address): [MarketplaceSaleOfferView] {
    let account = getAccount(address)
    let marketCollectionRef = account
        .getCapability<&BasketballsMarket.Collection{BasketballsMarket.CollectionPublic}>(
            BasketballsMarket.CollectionPublicPath
        )
        .borrow()
        ?? panic("Could not borrow market collection from address")
    let basketballsCollectionRef = account
      .getCapability<&Basketballs.Collection{NonFungibleToken.CollectionPublic, Basketballs.BasketballsCollectionPublic}>(
        Basketballs.CollectionPublicPath
      )
      .borrow()
      ?? panic("Could not borrow basketballs collection from address")
    let saleOfferViews: [MarketplaceSaleOfferView] = []
    let offerIds: [UInt64] = marketCollectionRef.getSaleOfferIDs()
    for offerId in offerIds {
      let saleItem = marketCollectionRef.borrowSaleItem(saleItemID: offerId)
      let basketball = basketballsCollectionRef.borrowBasketball(id: saleItem!.saleItemID)!
      let editionMetadata = Basketballs.getEditionMetadata(editionID: basketball.editionID)
      let saleOfferView = MarketplaceSaleOfferView(
        saleItemID: saleItem!.saleItemID,
        salePrice: saleItem!.salePrice,
        saleCompleted: saleItem!.saleCompleted,
        editionID: editionMetadata.editionID,
        serialNumber: basketball.serialNumber,
        name: editionMetadata.name,
        description: editionMetadata.description,
        imageURL: editionMetadata.imageURL,
        nftCount: editionMetadata.nftCount
      )
      saleOfferViews.append(saleOfferView)
    }
    return saleOfferViews
}

Describe the feature few words

Implement API changes from the redesign and optimization work.

What will be the end result of this change?

Changed GraphQL APIs that will provide better interactions with the playground and support new features.

Is there any previous work that needs to be completed before making this change?

• Redesign and optimise APIs #92
• Implement file entity #72

Which components will this change affect?
The frontend API implementation.

Describe the bug

Attempting to deploy Fungible Token contract from tutorial results in the following error:

To Reproduce

• Visit https://play.staging.onflow.org
• Paste the Fungible Token contract from here, and try to deploy it.:
  https://docs.onflow.org/cadence/tutorial/03-fungible-tokens#interacting-with-the-fungible-token-in-the-flow-playground

Expected behaviour
The contract deploys without errors.

Additional context
This is a regression. Currently blocking the Playground update from Cadence 0.12 to 0.14

Describe the bug
Contract name does not have a limit on cadence however I'm unable to deploy one with a name length of 9000 on playground.

To Reproduce
Steps to reproduce the behavior:
https://play.onflow.org/7ec2d192-c897-489e-a34b-d11f67507ea8?type=account&id=0

error says:

Error: GraphQL error: failed to update account: datastore: string property too long to index at index 0 for a Property with Name "DeployedContracts"

Expected behavior
Able to deploy the contract

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

We should periodically clear the database of outdated projects that are no longer used and were not marked as persisted. Those projects could be safely discarded after a few months. We should do some discovery on the reasonable interval.

We will need to change the API endpoints to accommodate the changes from the refactoring of accounts and contracts. Furthermore at this point, we should optimize those endpoints as currently there are always requests done in pairs by the client (deploy contract, get project to update). Those requests should have required data in response in the first place to optimize round trips.

When opening the playground tutorial for creating an NFT directly from the “click here to start a tutorial” button in the playground, there’s no Project name, details, or link to the docs tutorial.

Also, there seems to be 2 versions of the CreateNFT Tutorial (one of them having a blank landing project page).

1. From the “click here to start a tutorial” button > Create Non-Fungible Tokens > Open Project = As shown above, blank project.
2. From the “click here to start a tutorial” button > Create Non-Fungible Tokens > Read More = Tutorial 04 Non Fungible Token. From the tutorial page, you can also open the Playground Project, which leads to a non-blank project landing page (Cadence Non-Fungible Token Tutorial)

Therefore, the fix could be replacing the blank link from 1 to the project landing link in 2.

• Limit each user to creating at most 10 projects

Is your feature request related to a problem? Please describe.
A on-call runbook is needed to cover topics including but not limited to:

deployment process (already documented; ping kerry to link it)
when the app is down, how do we get started to debug
ensure everyone in the team has permissions needed to perform dev-ops

Currently, playground API doesn't support requests from Vercel deployed frontend, making it harder to test new features. Staging environment is also not supported by CORS.

DOD:

• Implement the support
• Deploy to staging

Describe the feature in a few words

We need to create custom errors to show the user which hides the backend details.

What will be the end result of this change?

The result of this change will allow the frontend to only display error messages that the user can understand and act upon. For example, database errors should just the show user "Something went wrong, please try again later". The details should however be piped into Sentry.

Is there any previous work that needs to be completed before making this change?

No previous work is required, but refactors will likely help determine the structure of errors.

Which components will this change affect?

This change will effect our Middlewares for Error handling and Sentry reporting.

Currently the code is organized and connected roughly as follows:

This is confusing with poor abstraction. The server is doing too much work (look at the main function).
Also, currently most modules are in top-level folders which makes control flow difficult to understand.

Here are my suggestions on how to improve the architecture:

The implementation is relatively straight forward, and won't take long. This will improve maintainability, and abstract functionality into a more modular approach.

Describe the feature few words

Change the implementation of contracts, transaction templates, and script templates to be a file.

What will be the end result of this change?

The end result of this change will enable the front end and backend to treat all user code as a file rather than separate entities. This will likely simplify the backend code, and provide a cleaner API.
General file endpoints include:

• Create new file (blank or from template)
• Update file name
• Delete file

Is there any previous work that needs to be completed before making this change?

This change requires the models and schema to be reworked first.

Which components will this change affect?

This change will effect the following components:

• Contract model
• Transaction template model
• Script template model
• GraphQL Schema

Other remarks

This change is a large refactor which will effect the underlaying functionality of all backend functions relating to executing contracts, scripts, and transactions on the blockchain.

The Playground dependencies are outdated. This should be fixed by upgrading all the dependencies and migrating the API changes. The bigger changes are coming from GQL library and a WIP PR is also made that can be continued here: #27

The Go version should also be updated to the latest 1.18. There's been some work already done to achieve this: #32

Fix linting issues

There were a lot of issues that were coming from the fact our current test doesn't test against the environment with replicas.

We need to create a couple of E2E tests that test against such an environment. My suggestion is to create a couple of tests in a setup where you spawn up two servers and create a simple random load balancer between them. Do some discovery if there's any existing work or practices to do that in Go.

There might even be a better approach by just creating two instances of existing client

I would then test just couple of scenarios:

• Executing couple of transactions, but distribute the requests between replicas
• Deploying couple of contract, but distribute the requests between replicas
• Deploy contract but redeploy on another replica

Describe the feature few words

This feature will enable the frontend to ask for a list of projects for a given user.

What will be the end result of this change?

The end result of this change will enable the front end to easily implement their project list user view.

• Enforce maximum limit of 10 projects per user

Is there any previous work that needs to be completed before making this change?

No previous work is required.

Which components will this change effect?

• This change will effect the backend API by providing more functionality.
• Schema additional query for projectList
• Schema additional mutation for deleteProject

Look into https://gqlgen.com/reference/dataloaders/ for optimizing requests to the database.

Setting up Sentry for the Backend integration. Front-end: onflow/flow-playground#252

Currently, we have a middleware that reports errors to the user as GraphQL errors. I think we should revisit this logic, especially to not leak implementation details to the user as they provide no value. I believe we should default to a default error message, we should anyhow get those errors as warnings in sentry. I would even further filter the messages to the Sentry to not include any errors with wrong user inputs etc, but just contain errors we actually care about.

Describe the feature few words

All API endpoints must be refactored to handle new models and functionality.

What will be the end result of this change?

This change will result in a new API with added functionality to support frontend needs. Furthermore, we should optimize those endpoints as currently there are always requests done in pairs by the client (deploy contract, get project to update). Those requests should have required data in response in the first place to optimize round trips.

Is there any previous work that needs to be completed before making this change?

New models must be defines before this refactor can take place.

Which components will this change affect?

This change will effect the following components:

• Updating storage of models
• Updating deployments, transactions, and script executions as files
• Updating data retrieval for projects and corresponding subcomponents

Other remarks

Describe the feature few words

Only one client request should be required for each API endpoint. Redesign the API to optimize the requests but also allow new API features.

What will be the end result of this change?

We should optimize endpoints as currently there are always requests done in pairs by the client (deploy contract, get project to update). Those requests should have required data in response in the first place to optimize round trips.

Is there any previous work that needs to be completed before making this change?

Previous work includes the models refactor.

Which components will this change affect?

This change will effect the client calling the API, as well as the API functions which make changes such that the frontend requires retrieving the changed data immediately after.

Add testing for the following controllers:

• acounts
• scripts
• transactions
• utils

API for reseting the project state, it could just be part of the project API

Describe the feature few words

Add an API endpoint for reseting the project state

What will be the end result of this change?

We can reset the project state

Is there any previous work that needs to be completed before making this change?

N/A

Which components will this change effect?

• API schema

We need to implement a history API that will provide information about:

• transaction executions
• script executions
• block creations

Also, we will need to augment that data with additional context like name of the transaction, contract, events etc. Needs to be further deinfed.

There are some state recreation issues as seen in Sentry:

https://sentry.io/organizations/dapperlabs/issues/3628363044/?project=6398442&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D&sort=inbox&statsPeriod=14d

https://sentry.io/organizations/dapperlabs/issues/3628608270/?project=6398442&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D&sort=inbox&statsPeriod=14d

https://sentry.io/organizations/dapperlabs/issues/3627952185/?project=6398442&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D&sort=inbox&statsPeriod=14d

https://sentry.io/organizations/dapperlabs/issues/3627949338/?project=6398442&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D&sort=inbox&statsPeriod=14d

Currently the playground API contains it's own implementation of blockchain execution in the compute module: https://github.com/onflow/flow-playground-api/tree/master/compute

This is not ideal as it adds more code to maintain but also exposes blockchain-specific domain to the maintainers of the playground, requiring much higher knowledge required. We should refactor the playground to include the emulator codebase for state execution.

Playground api needs to be updated to v0.20.0.

It seems trivial with upgrading dependancies and adding few functions to context ( they don't need to be implemented afaik )

with onflow/cadence#1251 this will allow playground to run at v0.20.0 latest version.

I can send a PR if needed.

• Create time-limited cache for project locks with 1 day timeout to avoid memory a leak type scenario

Describe the feature few words

Create an API endpoint which creates an empty project with no initial accounts or files.

What will be the end result of this change?

The end result will enable the front end to create a blank project with no initial accounts or code templates.

Is there any previous work that needs to be completed before making this change?

No previous work is needed.

Which components will this change affect?

This will only effect the API through adding new functionality.

Currently, content adaptors just find addresses with simple regex detecting addresses, which finds also addresses in strings and change them, making the logs misbehave if they have some logs targeting the address.

An example problematic script content:

import Foo from 0x01 // should be replaced
getAccount(0x02) // should be replaced
log("hey 0x02") // shouldn't be replaced

Improve the code that handles that, but FIRST write a test suite that asserts functioning and confirms the bug and then do the change. It's really important to not break the current implementation. Also, don't go down the path of using the Cadence parser because I did that before but the problem becomes how the addresses are stored in the parser which doesn't match source addresses so you will have no way of replacing them in source code.

Describe the feature few words

Change the API schema to handle cadence files instead of contracts, scripts, and transactions separately

What will be the end result of this change?

Simplified API schema and less copy pasted code on the backend.

• Playground tests should be simplified as well to just test file templates as a whole rather than handle each one separately.

Is there any previous work that needs to be completed before making this change?

File implementation refactor

Which components will this change effect?

• API schema for frontend
• Resolver simplification