baking-bad / tzkt Goto Github PK

View Code? Open in Web Editor NEW

181.0 12.0 34.0 4.83 MB

😼 Awesome Tezos blockchain indexer and API

Home Page: https://tzkt.io

License: MIT License

C# 99.92% Makefile 0.06% Dockerfile 0.02%

blockchain-indexer blockchain-api tezos-indexer tezos-blockchain tezos-api tzkt-indexer tzkt-api

tzkt's People

Stargazers

Watchers

tzkt's Issues

Discrepency between tzstats and tzkt

Hi all,

I wanted to point out that the test in TRD to compared the rewards between tzstats and tzkt fails since the transition happend.
tezos-reward-distributor-organization/tezos-reward-distributor#667

Test: test_get_rewards_for_cycle_map

Maybe you could investigate here?

Best regards
Carlo

Asking what is the additional factors to increase `totalSupply` in statistics api.

Hello tzkt team.

I have wonder what is the additional factors to increase totalSupply in statistics api.

Background:

Block height 1409023 's totalSupply return is 871244017824382.
That is the expected value.
Block height 1409023 's totalSupply return is 871244097675882.

Both block's total supply value difference is 79851500.
And endorsement reward is 40000000.
So left amount is 39851500.

I just tried to assemble this value by myself. But hard to understand.
Below is the return that I got from the indexer, If someone guide me would be so much appreciated.

1.6.0: Timeout and the requested operation requires an element of type 'Array', but the target element has type 'Object'.

Hi. Sync 1.6.0

info: Tzkt.Sync.Program[0]
      Initialize database
info: Tzkt.Sync.Program[0]
      Database initialized
warn: Tzkt.Sync.Services.Observer[0]
      Observer started
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
info: Tzkt.Sync.Services.Observer[0]
      State initialized: [1589246:BL2DKZ6CZfzG9wMjVCttdt5wK8fRydDvHXyVDnt3XJzEr2uiG5r]
info: Tzkt.Sync.Services.QuotesService[0]
      Quote provider: TzktQuotesProvider (Async)
info: Tzkt.Sync.Services.Observer[0]
      Quotes initialized: [1589246]
info: Tzkt.Sync.Services.Observer[0]
      Synchronization started
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The requested operation requires an element of type 'Array', but the target element has type 'Object'.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]

dotnet publish stuck at project restore

i was trying the tezos explorer setup from source but it is stuck at dotnet publish command

dotnet publish -o ~/tzkt-sync
MSBuild version 17.7.4+3ebbd7c49 for .NET
Determining projects to restore...

How can i resolve this? . Thanks in advance

Not issuing is our key to keep protecting safe progressions*!

Create a new content is not issuing (damaging) the "baking-bad", rather progressing new means to FREE BADNESS (ignorance). Share basic/simple truths about yourself! ThanksGiving-MoR_e-nergy!

Indexer stops at block 2441748

Hi,

I have two instances of tzkt (v1.8.4) running against tezos-node v13.0 (mainnet).

Both tzkt indexers stopped at the same block. The current blockheight reported by the tzkt api is 2441748.
Indexer log:

         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
fail: Tzkt.Sync.Services.Observer[0]
      Failed to rebase branch. An exception has been raised that is likely due to a transient failure.
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Tzkt.Sync.Protocols.Proto1.BigMapCommit.Revert(Block block) in /app/src/tzkt/Tzkt.Sync/Protocols/Handlers/Proto1/Commits/BigMapCommit.cs:line 527
         at Tzkt.Sync.Protocols.Proto12Handler.Revert() in /app/src/tzkt/Tzkt.Sync/Protocols/Handlers/Proto12/Proto12Handler.cs:line 315
         at Tzkt.Sync.ProtocolHandler.RevertLastBlock(String predecessor) in /app/src/tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 141
         at Tzkt.Sync.ProtocolHandler.RevertLastBlock(String predecessor) in /app/src/tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 161
         at Tzkt.Sync.Services.Observer.RebaseLocalBranchAsync(CancellationToken cancelToken) in /app/src/tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 157
         at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/src/tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 74
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply block: Validation exception - invalid block predecessor. Rebase local branch...
fail: Tzkt.Sync.Services.Observer[0]
      Invalid head [2441747:BLMGwtZ7MRT28SpJbs1XGJGnSHQqWH4tJB8K2FHZfbhthLvJT4d]. Reverting...

I tried to restart them but it did not help.

Please add Prometheus metrics to services

Prometheus is the leading open source metrics collection app and has countless popular integrations with other apps.

Currently i can only monitor the tzkt apps as part of my docker monitoring stack and only get very limited metrics from that. My request is if it would be possible to expose some or even basic Prometheus metrics that will give some idea about the health of the app (e.g. is the sync running, how far behind it is compared to latest block, time since latest block etc.)

Thanks

Please add API endpoints for current and historical prices of FA1.2 and FA2 tokens

Please make it possible to get the current and historical prices of FA1.2 and FA2 tokens. e.g.
User holds 1000 Crunch and bought 20.02.22 at X price. And current price is XY.
So you can get individual token prices or aggregate prices over time and have price history etc.
With growing Tezos ecosystem this is a big piece thats missing in tzkt.

Thank you!

Slashing of unstaked deposits cannot be implemented due to bugs in Oxford

We started getting this error this morning in both our dev and prod environment. I am hoping to find some guidence on how to resolve the issue. Thank you in advance.

dbug: Tzkt.Sync.Services.Observer[0]
      New head is found [5651496:BKpWMvJwGFGtzksZVoUDvm4bMMSaYeuyiNF81x3ahhQ5btm3k37]
dbug: Tzkt.Sync.Services.Observer[0]
      Applying block...
dbug: Tzkt.Sync.Protocols.Proto18Handler[0]
      Load block 5650612
dbug: Tzkt.Sync.Protocols.Proto18Handler[0]
      Begin DB transaction
dbug: Tzkt.Sync.Protocols.Proto18Handler[0]
      Warm up cache
dbug: Tzkt.Sync.Protocols.Proto18Handler[0]
      Validate block
dbug: Tzkt.Sync.Protocols.Proto18Handler[0]
      Process block
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates
      System.NotImplementedException: Slashing of unstaked deposits cannot be implemented due to bugs in Oxford. Let's wait for fixes...
         at Tzkt.Sync.Protocols.Proto18.StakingCommit.Apply(Block block, JsonElement op, JsonElement content) in /usr/src/tzkt/Tzkt.Sync/Protocols/Handlers/Proto18/Commits/Operations/StakingCommit.cs:line 277
         at Tzkt.Sync.Protocols.Proto18Handler.Commit(JsonElement block) in /usr/src/tzkt/Tzkt.Sync/Protocols/Handlers/Proto18/Proto18Handler.cs:line 180
         at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /usr/src/tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 77
         at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /usr/src/tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 138
         at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /usr/src/tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 183
         at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /usr/src/tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 66

Error Message: https://ghostnet.tzkt.io/

Hello Team,

I have an error (screenshot) that does not seem to have impact on the functionality of the website (not that I can tell ;) ):

It seems to be the result of a firewall block of dipdup.io:

Suggestion:
If a user is asking IT to allow access to a website maybe the website should route calls do other entities through an alias (is this possible). Instead of using the not accessible dipdup.net website maybe the original link of tzkt.io?

tzkt.io (website) loses the URL in Chrome and Brave

This is a bug in the website tzkt.io, but I can't find a better place to report it.

Bug: The tzkt.io website periodically removes the URL suffix in Chrome and Brave.

E.g., suppose I have a tab open to https://tzkt.io/KT1TxqZ8QtKvLu3V3JH7Gx58n7Co8pgtpQU5/operations/
If I restart Chrome, there's a decent chance the tab will now point to https://tzkt.io (no suffix).
This also seems to happen sometimes even if I don't restart Chrome.

Why it matters: I like to have tabs open to different addresses I'm interested in tracking, and this breaks that workflow.

Accept protocol Alpha

In Taqueria, we can start the Flextesa sandbox in protocol Alpha with identifier: ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK.

Is it possible to accept this as a valid protocol? I understand that there might be features in protocol Alpha that are not yet implemented in TzKt, but we can let the user accept that risk and run their own TzKt instance?

Function call args types

Hi!

Can tzkt use convert args types?

Currently:

{
    "parameter": {
        "entrypoint": "mint",
        "value": {
            "_to": "tz1Yigc57GHQixFwDEVzj5N1znSCU3aq15td",
            "value": "11"
        }
    },
}

But with parameters of:

    "parameters": {
        "entrypoint": "mint",
        "value": {
            "prim": "Pair",
            "args": [
                {
                    "string": "tz1Yigc57GHQixFwDEVzj5N1znSCU3aq15td"
                },
                {
                    "int": "22"
                }
            ]
        }
    },

Shouldn't it has value as a JSON int like this?

{
    "parameter": {
        "entrypoint": "mint",
        "value": {
            "_to": "tz1Yigc57GHQixFwDEVzj5N1znSCU3aq15td",
            "value": 11
        }
    },
}

Looking for a way to know if a delegate has upvoted for proposals, in proposal period

I'm looking for a way to know if a delegate has upvoted for proposals, in proposal period.

In short, I want to know if a delegate is in this list: https://tzkt.io/governance/41/proposal

For ballot vote, we have GET ../<block_id>/votes/ballot_list on tezos-client. Anything similar for proposals upvoting ?

Lima now prevents replay of governance proposals (gitlab.com/tezos/tezos/-/merge_requests/5828). I assume this means that the proposals per baker are now stored in the context and there could be a RPC for it. But I don't think there is such a RPC now?

https://tezos.stackexchange.com/questions/4452/proposal-upvotes-list

Test

failed to synchronization.

Node : https://rpc.tzkt.io/mainnet/

Logs

info: Tzkt.Sync.Program[0]
      Version 1.8.3.0
info: Tzkt.Sync.Program[0]
      Initialize database
info: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 5.0.10 initialized 'TzktContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: None
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (10ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid=c.relnamespace WHERE c.relname='__EFMigrationsHistory');
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT "MigrationId", "ProductVersion"
      FROM "__EFMigrationsHistory"
      ORDER BY "MigrationId";
info: Tzkt.Sync.Program[0]
      Database initialized
warn: Tzkt.Sync.Services.Observer[0]
      Observer started
info: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 5.0.10 initialized 'TzktContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: None
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (6ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."AccountCounter", a."AccountsCount", a."ActivationOpsCount", a."BallotOpsCount", a."BigMapCounter", a."BigMapKeyCounter", a."BigMapUpdateCounter", a."BlocksCount", a."Chain", a."ChainId", a."CommitmentsCount", a."ConstantsCount", a."Cycle", a."CyclesCount", a."DelegationOpsCount", a."DoubleBakingOpsCount", a."DoubleEndorsingOpsCount", a."DoublePreendorsingOpsCount", a."EndorsementOpsCount", a."EndorsingRewardOpsCount", a."Hash", a."KnownHead", a."LastSync", a."Level", a."ManagerCounter", a."Metadata", a."MigrationOpsCount", a."NextProtocol", a."NonceRevelationOpsCount", a."OperationCounter", a."OriginationOpsCount", a."PreendorsementOpsCount", a."ProposalOpsCount", a."ProposalsCount", a."Protocol", a."ProtocolsCount", a."QuoteBtc", a."QuoteCny", a."QuoteEth", a."QuoteEur", a."QuoteGbp", a."QuoteJpy", a."QuoteKrw", a."QuoteLevel", a."QuoteUsd", a."RegisterConstantOpsCount", a."RevealOpsCount", a."RevelationPenaltyOpsCount", a."ScriptCounter", a."SetDepositsLimitOpsCount", a."StorageCounter", a."Timestamp", a."TokenBalanceCounter", a."TokenBalancesCount", a."TokenCounter", a."TokenTransfersCount", a."TokensCount", a."TransactionOpsCount", a."VotingEpoch", a."VotingPeriod"
      FROM "AppState" AS a
      LIMIT 2
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /opt
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."ActiveTokensCount", a."Address", a."Balance", a."ContractsCount", a."Counter", a."DelegateId", a."DelegationLevel", a."DelegationsCount", a."FirstLevel", a."LastLevel", a."Metadata", a."MigrationsCount", a."OriginationsCount", a."RevealsCount", a."Staked", a."TokenBalancesCount", a."TokenTransfersCount", a."TransactionsCount", a."Type", a."Activated", a."PublicKey", a."RegisterConstantsCount", a."Revealed", a."SetDepositsLimitsCount", a."ActivationLevel", a."BallotsCount", a."BlocksCount", a."DeactivationLevel", a."DelegatedBalance", a."DelegatorsCount", a."DoubleBakingCount", a."DoubleEndorsingCount", a."DoublePreendorsingCount", a."EndorsementsCount", a."EndorsingRewardsCount", a."FrozenDeposit", a."FrozenDepositLimit", a."NonceRevelationsCount", a."PreendorsementsCount", a."ProposalsCount", a."RevelationPenaltiesCount", a."SoftwareId", a."StakingBalance"
      FROM "Accounts" AS a
      WHERE a."Type" = 1
info: Tzkt.Sync.Services.Observer[0]
      State initialized: [2368491:BLYA2gBkkYgGvrHQ1VdsSWkHtPLXeWtFgh33fXxnQKWRmGVn6oq]
info: Tzkt.Sync.Services.QuotesService[0]
      Quote provider: DefaultQuotesProvider (Async)
info: Tzkt.Sync.Services.Observer[0]
      Quotes initialized: [2368491]
info: Tzkt.Sync.Services.Observer[0]
      Synchronization started
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]

I ran indexer & indexer-api altogether in the same container.

Issue with ithacanet and Tzkt.Sync

Hello,

I'm trying to deploy a private tzkt on ithacanet but i have an error. My setup is:

a private tezos archive node (ithacanet)
a Aurora Postgresql 13.4
tzkt.Sync and tzkt.Api deployed on kubernetes

The pod starts fine and begin to synchronize and then i have this message in the tzkt.Sync logs:

info: Tzkt.Sync.Services.Observer[0]
      Applied 312193 of 548025
info: Tzkt.Sync.Services.Observer[0]
      Applied 312194 of 548025
info: Tzkt.Sync.Services.Observer[0]
      Applied 312195 of 548025
info: Tzkt.Sync.Services.Observer[0]
      Applied 312196 of 548025
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata

My Tezos node runs fine (sync is ok)

$ tezos-client rpc get /monitor/bootstrapped
Warning:
  
                 This is NOT the Tezos Mainnet.
  
           Do NOT use your fundraiser keys on this network.

{ "block": "BL88BSbBhs8eb4M2gMYSG5WFMgKn1owsXAW2Mk5UfhwSNXRbrVS",
  "timestamp": "2022-05-18T08:24:55Z" }

~ $ tezos-client bootstrapped
Warning:
  
                 This is NOT the Tezos Mainnet.
  
           Do NOT use your fundraiser keys on this network.

Node is bootstrapped.

Nothing in the Aurora logs.

Could you please help me to debug this ?
Thanks.

Regards,

Division by zero

Hi!

Getting this exception like a minute after starting the service:

tzkt-sync  | fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
tzkt-sync  |       Failed executing DbCommand (1ms) [Parameters=[@__p_0='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
tzkt-sync  |       SELECT b."Id", b."BlockRound", b."Bonus", b."Cycle", b."Deposit", b."Events", b."Fees", b."Hash", b."LBEscapeEma", b."LBEscapeVote", b."Level", b."Metadata", b."Operations", b."PayloadRound", b."ProducerId", b."ProposerId", b."ProtoCode", b."ResetBakerDeactivation", b."ResetProposerDeactivation", b."RevelationId", b."Reward", b."SoftwareId", b."Timestamp", b."Validations", t."Id", t."ActiveTokensCount", t."Address", t."Balance", t."ContractsCount", t."Counter", t."DelegateId", t."DelegationLevel", t."DelegationsCount", t."FirstLevel", t."LastLevel", t."Metadata", t."MigrationsCount", t."OriginationsCount", t."RevealsCount", t."Staked", t."TokenBalancesCount", t."TokenTransfersCount", t."TransactionsCount", t."Type", t."Activated", t."PublicKey", t."RegisterConstantsCount", t."Revealed", t."SetDepositsLimitsCount", t."ActivationLevel", t."BallotsCount", t."BlocksCount", t."DeactivationLevel", t."DelegatedBalance", t."DelegatorsCount", t."DoubleBakingCount", t."DoubleEndorsingCount", t."DoublePreendorsingCount", t."EndorsementsCount", t."EndorsingRewardsCount", t."FrozenDeposit", t."FrozenDepositLimit", t."NonceRevelationsCount", t."PreendorsementsCount", t."ProposalsCount", t."RevelationPenaltiesCount", t."SoftwareId", t."StakingBalance", p."Id", p."BallotQuorumMax", p."BallotQuorumMin", p."BlockDeposit", p."BlockReward0", p."BlockReward1", p."BlocksPerCommitment", p."BlocksPerCycle", p."BlocksPerSnapshot", p."BlocksPerVoting", p."ByteCost", p."Code", p."ConsensusThreshold", p."DoubleBakingPunishment", p."DoubleEndorsingPunishmentDenominator", p."DoubleEndorsingPunishmentNumerator", p."EndorsementDeposit", p."EndorsementReward0", p."EndorsementReward1", p."EndorsersPerBlock", p."FirstCycle", p."FirstCycleLevel", p."FirstLevel", p."FrozenDepositsPercentage", p."HardBlockGasLimit", p."HardOperationGasLimit", p."HardOperationStorageLimit", p."Hash", p."LBEscapeThreshold", p."LBSubsidy", p."LBSunsetLevel", p."LastLevel", p."MaxBakingReward", p."MaxEndorsingReward", p."MaxSlashingPeriod", p."Metadata", p."MinParticipationDenominator", p."MinParticipationNumerator", p."NoRewardCycles", p."OriginationSize", p."PreservedCycles", p."ProposalQuorum", p."RampUpCycles", p."RevelationReward", p."TimeBetweenBlocks", p."TokensPerRoll"
tzkt-sync  |       FROM "Blocks" AS b
tzkt-sync  |       INNER JOIN "Protocols" AS p ON b."ProtoCode" = p."Code"
tzkt-sync  |       LEFT JOIN (
tzkt-sync  |           SELECT a."Id", a."ActiveTokensCount", a."Address", a."Balance", a."ContractsCount", a."Counter", a."DelegateId", a."DelegationLevel", a."DelegationsCount", a."FirstLevel", a."LastLevel", a."Metadata", a."MigrationsCount", a."OriginationsCount", a."RevealsCount", a."Staked", a."TokenBalancesCount", a."TokenTransfersCount", a."TransactionsCount", a."Type", a."Activated", a."PublicKey", a."RegisterConstantsCount", a."Revealed", a."SetDepositsLimitsCount", a."ActivationLevel", a."BallotsCount", a."BlocksCount", a."DeactivationLevel", a."DelegatedBalance", a."DelegatorsCount", a."DoubleBakingCount", a."DoubleEndorsingCount", a."DoublePreendorsingCount", a."EndorsementsCount", a."EndorsingRewardsCount", a."FrozenDeposit", a."FrozenDepositLimit", a."NonceRevelationsCount", a."PreendorsementsCount", a."ProposalsCount", a."RevelationPenaltiesCount", a."SoftwareId", a."StakingBalance"
tzkt-sync  |           FROM "Accounts" AS a
tzkt-sync  |           WHERE a."Type" = 1
tzkt-sync  |       ) AS t ON b."ProposerId" = t."Id"
tzkt-sync  |       WHERE (((b."Level" % p."BlocksPerCommitment") = 0) AND (b."Cycle" = @__p_0)) AND (b."RevelationId" IS NULL)
tzkt-sync  | fail: Microsoft.EntityFrameworkCore.Query[10100]
tzkt-sync  |       An exception occurred while iterating over the results of a query for context type 'Tzkt.Data.TzktContext'.
tzkt-sync  |       Npgsql.PostgresException (0x80004005): 22012: division by zero
tzkt-sync  |          at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
tzkt-sync  |          at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
tzkt-sync  |         Exception data:
tzkt-sync  |           Severity: ERROR
tzkt-sync  |           SqlState: 22012
tzkt-sync  |           MessageText: division by zero
tzkt-sync  |           File: int.c
tzkt-sync  |           Line: 1118
tzkt-sync  |           Routine: int4mod
tzkt-sync  |       Npgsql.PostgresException (0x80004005): 22012: division by zero
tzkt-sync  |          at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
tzkt-sync  |          at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
tzkt-sync  |         Exception data:
tzkt-sync  |           Severity: ERROR
tzkt-sync  |           SqlState: 22012
tzkt-sync  |           MessageText: division by zero
tzkt-sync  |           File: int.c
tzkt-sync  |           Line: 1118
tzkt-sync  |           Routine: int4mod
tzkt-sync  | fail: Tzkt.Sync.Services.Observer[0]
tzkt-sync  |       Failed to apply updates. 22012: division by zero
tzkt-sync  |       Npgsql.PostgresException (0x80004005): 22012: division by zero
tzkt-sync  |          at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
tzkt-sync  |          at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
tzkt-sync  |          at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
tzkt-sync  |          at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
tzkt-sync  |          at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
tzkt-sync  |          at Tzkt.Sync.Protocols.Proto2.RevelationPenaltyCommit.Apply(Block block, JsonElement rawBlock) in /app/Tzkt.Sync/Protocols/Handlers/Proto2/Commits/RevelationPenaltyCommit.cs:line 26
tzkt-sync  |          at Tzkt.Sync.Protocols.Proto8Handler.Commit(JsonElement block) in /app/Tzkt.Sync/Protocols/Handlers/Proto8/Proto8Handler.cs:line 45
tzkt-sync  |          at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 63
tzkt-sync  |          at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 103
tzkt-sync  |          at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 179
tzkt-sync  |          at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 64
tzkt-sync  |         Exception data:
tzkt-sync  |           Severity: ERROR
tzkt-sync  |           SqlState: 22012
tzkt-sync  |           MessageText: division by zero
tzkt-sync  |           File: int.c
tzkt-sync  |           Line: 1118
tzkt-sync  |           Routine: int4mod

Is it trying to index block 0 ? Pretty weird!

Any idea please?

Thanks!

How to connect to a local node

By default, tezos.giganode.io is used, but I want to connect to the local node, how should I modify it

TzKt connected to local flextesa returns the wrong Head info

I have connected TzKt to a local FlexTesa sandbox.
The local sandbox is at level 3000, but TzKt returns this for /v1/head:

{
  "chain": "private",
  "chainId": "NetXmGq7LPFBoxA",
  "cycle": 6,
  "level": 52,
  "hash": "BKrUrKmYYByPobctsS7WYYEJZqtmSXY5AA7nkzTjQA2tNqkoDxT",
  "protocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
  "nextProtocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
  "timestamp": "2022-08-17T23:31:22Z",
  "votingEpoch": 3,
  "votingPeriod": 3,
  "knownLevel": 52,
  "lastSync": "2022-08-17T16:31:22Z",
  "synced": true,
  "quoteLevel": 52,
  "quoteBtc": 0.00008064180433753636,
  "quoteEur": 1.848423050861913,
  "quoteUsd": 1.8808259198083772,
  "quoteCny": 12.753880562220555,
  "quoteJpy": 254.14848324002614,
  "quoteKrw": 2472.9213754747293,
  "quoteEth": 0.0010261347976775258,
  "quoteGbp": 1.5616290721317734
}

When looking at the database, I see that all blocks are indexed and new blocks keep getting indexed. Same is evident from TzKt.Sync logs.

What can be the cause of this behaviour?

Allow overriding the number of attempts in TzKt.Api

In Tzkt.Api/Program.cs We have a hardcoded attempt = 30;.

In Taqueria, we start a Flextesa Sandbox, then a TzKt.Sync, postgres, and finally a TzKt.Api. Some times, the hardcoded 30 attempts is not enough and Api exits before Sync can update the database.

Can we Allow overriding this value from config or environment variables?

Protocol invoice for Florence activation done at wrong height

We run a Tezos node and tzkt instance and noticed the following:

According to the node, tz1abmz7jiCV2GH2u81LRrGgAFFgvQgiDiaf gets 100 XTZ at level 1466368

$ curl --silent "http://localhost:8832/chains/NetXdQprcVkpaWU/blocks/1466368"
... 
            {
                "kind": "contract",
                "contract": "tz1abmz7jiCV2GH2u81LRrGgAFFgvQgiDiaf",
                "change": "100000000",
                "origin": "migration"
            },
...

But according to tzkt, it happens at level 1466367:

$ curl --silent "http://localhost:5000/v1/blocks/1466367?operations=true" | jq ".migrations"
[
  {
    "type": "migration",
    "id": 52106114367488,
    "level": 1466367,
    "timestamp": "2021-05-11T01:30:26Z",
    "block": "BKrZu79PEc3UkmseSh7NQdRyKY91r3YWWPte8pcLiwk2aVkrnTE",
    "kind": "proposal_invoice",
    "account": {
      "address": "tz1abmz7jiCV2GH2u81LRrGgAFFgvQgiDiaf"
    },
    "balanceChange": 100000000
  }
]

kUSD balance changed to zero after Florence

We got a report from our user that his wallet balance is shown as zero after upgrade to Florence:
https://gitlab.com/camlcase-dev/dexter-frontend/-/issues/6

It does not apply to all wallets, not sure why.

Add filter by vote for ballot operations

I wanna be able to make requests like:

https://api.tzkt.io/v1/operations/ballots?period=57&proposal=PtHangzHogokSuiMHemCuowEavgYTP8J5qQ9fQS793MHYFpCY3r&vote=nay

https://api.tzkt.io/v1/operations/ballots?period=57&proposal=PtHangzHogokSuiMHemCuowEavgYTP8J5qQ9fQS793MHYFpCY3r&vote.in=nay,yay

Update sampler to match tezos/tezos

I recently fixed a minor bug in the corresponding tezos code here: https://gitlab.com/tezos/tezos/-/merge_requests/4385

It looks like tzkt still needs an update to do the same:

tzkt/Tzkt.Sync/Protocols/Helpers/Sampler.cs

Line 109 in a284db8

var r = Math.Abs(state.ReadInt64(pos));

Readiness and Liveliness Probes

My team and I really love this indexer and appreciate the hard work of the Baking Bad team. One thing that I think might be mutually beneficial to everyone is to add readiness and liveliness probes in the synchronizer/API so that users can create custom alerting and handling Kubernetes logic in the case of error.

Please add an API endpoint for historical token balances

Please make it possible to get historical token balances, e.g. "how many tzBTC did I have at Level N?".
This would really help accounting.
Thanks!

Add support for Jakarta

Hi team,

I would like to use Kukai wallet for Jakarta demos but apprently they have a dependency on your indexer

kukai-wallet/kukai#345

Do you have any plan to support Jakarta soon ?

Thank you

Failed to initialize database when running indexer for testnet

I am trying to run the indexer for delphinet as explained at this step. However, I am stuck at this error. I don't understand the error message:

$ dotnet Tzkt.Sync.dll
info: Tzkt.Sync.Program[0]
      Initialize database
crit: Tzkt.Sync.Program[0]
      Failed to initialize database: Resource temporarily unavailable
Unhandled exception. System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
   at System.Net.Dns.InternalGetHostByName(String hostName)
   at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
   at Npgsql.NpgsqlConnector.Connect(NpgsqlTimeout timeout)
   at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnection.Open()
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists()
   at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.Exists()
   at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.GetAppliedMigrations()
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.GetAppliedMigrations(DatabaseFacade databaseFacade)
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 60
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.IHostExt.Init(IHost host, Int32 attempt) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 83
   at Tzkt.Sync.Program.Main(String[] args) in /home/nguyen-van/tzkt/Tzkt.Sync/Program.cs:line 20
Aborted (core dumped)

Does this mean that it cannot have access to the database? Or to rpc.tzkt.io/delphinet? Yet, the database is available:

$ nmap -PN localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2020-12-29 12:05 CET
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000068s latency).
Not shown: 998 closed ports
PORT     STATE SERVICE
631/tcp  open  ipp
5432/tcp open  postgresql

Indexer got stuck in reverting status

Hi,

Somehow our mainnet tzkt got stuck in reverting status and become out of sync...

bug when sync data

info: Program[0]
Version 1.13.3.0
info: Program[0]
Initialize database...
info: Program[0]
Database initialized
warn: Tzkt.Sync.Services.Observer[0]
Observer started
warn: Microsoft.AspNetCore.Server.Kestrel[0]
Overriding address(es) 'http://+:80'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://0.0.0.0:5001
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: /app
info: Tzkt.Sync.Services.Observer[0]
State initialized: [5650611:BLk9432iUpkb21WPUmFQkqeBsUBT1ecsa9836d95PxzU4BvHPJ9]
info: Tzkt.Sync.Services.QuotesService[0]
Quote provider: TzktQuotesProvider (Async)
info: Tzkt.Sync.Services.Observer[0]
Quotes initialized: [5650611]
info: Tzkt.Sync.Services.Observer[0]
Synchronization started
warn: Microsoft.EntityFrameworkCore.Query[10103]
The query uses the 'First'/'FirstOrDefault' operator without 'OrderBy' and filter operators. This may lead to unpredictable results.
fail: Tzkt.Sync.Services.Observer[0]
Failed to apply updates
System.NotImplementedException: Slashing of unstaked deposits cannot be implemented due to bugs in Oxford. Let's wait for fixes...
at Tzkt.Sync.Protocols.Proto18.StakingCommit.Apply(Block block, JsonElement op, JsonElement content) in /app/Tzkt.Sync/Protocols/Handlers/Proto18/Commits/Operations/StakingCommit.cs:line 277
at Tzkt.Sync.Protocols.Proto18Handler.Commit(JsonElement block) in /app/Tzkt.Sync/Protocols/Handlers/Proto18/Proto18Handler.cs:line 180
at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 77
at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 138
at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 183
at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 66
fail: Tzkt.Sync.Services.Observer[0]
Failed to apply updates
System.NotImplementedException: Slashing of unstaked deposits cannot be implemented due to bugs in Oxford. Let's wait for fixes...
at Tzkt.Sync.Protocols.Proto18.StakingCommit.Apply(Block block, JsonElement op, JsonElement content) in /app/Tzkt.Sync/Protocols/Handlers/Proto18/Commits/Operations/StakingCommit.cs:line 277
at Tzkt.Sync.Protocols.Proto18Handler.Commit(JsonElement block) in /app/Tzkt.Sync/Protocols/Handlers/Proto18/Proto18Handler.cs:line 180
at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 77
at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 138
at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 183
at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 66

Historical big_map API and contract big_map history return different values

If I try to access big map at a specific level using its name in /contracts/ API I get different results to what I get if I request historical data at the same level using the same big_maps API.
Here is an example:
https://api.tzkt.io/v1/bigmaps/2013/historical_keys/1649387
https://api.tzkt.io/v1/contracts/KT1FG63hhFtMEEEtmBSX2vuFmP87t9E7Ab4t/bigmaps/ledger/historical_keys/1649387

2013 is ledger big map id of contract KT1FG63hhFtMEEEtmBSX2vuFmP87t9E7Ab4t

You can see balances values are different in ways I can't comprehend.
Is it a bug? Which one is correct?

The`voting/proposals/` endpoint returns wrong status for proposal.

The Edo proposal is still in proposal phase, but its status is returned as skipped while using this endpoint. This is the response returned as of now.

{
  "hash": "PtEdoTezd3RHSC31mpxxo1npxFjoWWcFgQtxapi51Z8TLu6v6Uq",
  "initiator": {
    "alias": "Stakery",
    "address": "tz1go7f6mEQfT2xX2LuHAqgnRGN6c2zHPf5c"
  },
  "period": 37,
  "upvotes": 3331,
  "status": "skipped",
  "metadata": {
    "alias": "Edo",
    "agora": "https://www.tezosagora.org/proposal/9",
    "invoice": 0
  }
}

Protocol not supported issue

I was trying to setup tzkt explorer for my tezos sandbox node. but it says protocol not supported.

fail: Tzkt.Sync.Services.Observer[0]
Failed to apply updates
System.NotImplementedException: Protocol 'ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK' is not supported
at Tzkt.Sync.TezosProtocols.GetProtocolHandler(IServiceProvider services, Int32 level, String protocol) in /app/Tzkt.Sync/Protocols/TezosProtocols.cs:line 55
at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 84
at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 130
at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 183
at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 66

can't i use alpha protocol with tzkt ?

Tzkt 1.8.0, Tezos Mainnet: reached maximum value of sequence "BakingRights_Id_seq"

Hello,
i am running Tzkt for production.
Mainnet, node version 12.2.

Tonight tzkt indexer hits the limits on sequence (as i understand it):

fail: Tzkt.Sync.Services.Observer[0] Failed to apply updates. 2200H: nextval: reached maximum value of sequence "BakingRights_Id_seq" (2147483647)

Please let me know if i can gather additional debug information.
I am in situation when production service is down.

API and indexer is at version 1.8.0

Support for smart contracts and big maps

I am working on Tezos Reward Distributor and we are using tzkt as a possible backend api for fetching blockchain data. We are now working on adding support for delegations in the form of dexter contracts and we would like to extend the current tzkt provider implementation with the needed endpoints to extract data from big maps at any given block, i.e. given a block level and a big map id, I would like to get the list of values stored in this big map at that specific block.
That would be great if this feature can be added to tzkt for us to enable support for the dexter delegations using tzkt.

Invalid head [*******]. Reverting...

Hi. v1.9.3
One of the two sync modules crashes with this error. Last night, two falls at different heights. Restoring from a snapshot helps but not for long.

fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (30,008ms) [Parameters=[@__contract_Id_0='?' (DbType = Int32), @__storage_Id_1='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
      SELECT s."Id", s."ContractId", s."Current", s."JsonValue", s."Level", s."MigrationId", s."OriginationId", s."RawValue", s."TransactionId"
      FROM "Storages" AS s
      WHERE (s."ContractId" = @__contract_Id_0) AND (s."Id" < @__storage_Id_1)
      ORDER BY s."Id" DESC
      LIMIT 1
fail: Microsoft.EntityFrameworkCore.Query[10100]
      An exception occurred while iterating over the results of a query for context type 'Tzkt.Data.TzktContext'.
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
fail: Tzkt.Sync.Services.Observer[0]
      Failed to rebase branch. An exception has been raised that is likely due to a transient failure.
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Tzkt.Sync.Protocols.Proto1.TransactionsCommit.RevertStorage(TransactionOperation transaction) in /app/Tzkt.Sync/Protocols/Handlers/Proto1/Commits/Operations/TransactionsCommit.cs:line 614
         at Tzkt.Sync.Protocols.Proto1.TransactionsCommit.RevertInternal(Block block, TransactionOperation transaction) in /app/Tzkt.Sync/Protocols/Handlers/Proto1/Commits/Operations/TransactionsCommit.cs:line 488
         at Tzkt.Sync.Protocols.Proto13Handler.Revert() in /app/Tzkt.Sync/Protocols/Handlers/Proto13/Proto13Handler.cs:line 449
         at Tzkt.Sync.ProtocolHandler.RevertLastBlock(String predecessor) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 141
         at Tzkt.Sync.ProtocolHandler.RevertLastBlock(String predecessor) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 161
         at Tzkt.Sync.Services.Observer.RebaseLocalBranchAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 157
         at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 74
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply block: Validation exception - invalid block predecessor. Rebase local branch...
fail: Tzkt.Sync.Services.Observer[0]
      Invalid head [2564100:BMJqWTFCoqpCruWhBzqCPHPW5Zppk947cYb42ehrtVA5Av1qmth]. Reverting...

Please migrate to .net 7 (or at least 6)

We are using TzKt in Taqueria to index a local Flextesa sandbox. It runs on dewveloper's computer and works very well except:

It does not run on mac m1

I have migrated the project to .net 6 and it's working fine.
Additional changes to the CI will be needed to create the multi-architecture docker image. But I guess I can contribute that as well.

So the question is: is there any reason to stay on dotnet 5? Or if we contribute the change there's a high chance for it to be merged?

Thanks

Failed to apply updates. Missed required property metadata

Hi! First of all thanks for this project; it's great!

I'm trying to use the docker builds with compose. I have tried 1.6.4, 1.7.0, and the latest all with the same issue. I've wiped the data volume each time.

tkzt_sync keeps responding with this general error:

tzkt_sync.1.p3m9pxjvwbib@rimmer    | fail: Tzkt.Sync.Services.Observer[0]
tzkt_sync.1.p3m9pxjvwbib@rimmer    |       Failed to apply updates. Serialization exception - Missed required property metadata

I'm not sure what it is choking on. Perhaps it is because I'm using the newest tezos-node with the updated storage format?

tezos-node --version
6bc29438 (2022-01-11 16:35:24 +0100) (11.1)

docker --version
Docker version 20.10.6, build 370c289

I don't think it matters since I'm using docker but I'm running this on:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

Linux  5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Let me know if you need any more information, thank you.

Missing permissions with PostgreSQL ?

I am running Ubuntu 20.04 and I am stuck at this step of the setup. Here is the output:

$ sudo -u postgres pg_restore -c --if-exists -v -d tzkt_db -1 tzkt_db.backup
could not change directory to "/home/nguyen-van": Permission denied
pg_restore: error: could not open input file "tzkt_db.backup": No such file or directory

Maybe a step is missing in the previous instructions give the right access?

tzkt-sync:1.4 not sync

run option

"dotnet Tzkt.Sync.dll"

docker-entrypoint.sh
#!/bin/bash
sed -i 's/.*DefaultConnection.*/    "DefaultConnection": "server=localhost;port=5432;database=tzkt_db;123=tzkt;password=123;"/' /app/appsettings.json
exec "$@"

Log

info: Tzkt.Sync.Services.Observer[0]
      Applied 8184 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8185 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8186 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8187 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8188 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8189 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8190 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8191 of 1426980
info: Tzkt.Sync.Services.Observer[0]
      Applied 8192 of 1426980
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (1ms) [Parameters=[@__address_0='?' (DbType = StringFixedLength)], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."Address", a."Balance", a."ContractsCount", a."Counter", a."DelegateId", a."DelegationLevel", a."DelegationsCount", a."FirstLevel", a."LastLevel", a."MigrationsCount", a."OriginationsCount", a."RevealsCount", a."Staked", a."TransactionsCount", a."Type", a."CreatorId", a."Kind", a."ManagerId", a."Spendable", a."Tzips", a."WeirdDelegateId", a."Activated", a."PublicKey", a."Revealed", a."ActivationLevel", a."BallotsCount", a."BlocksCount", a."DeactivationLevel", a."DelegatorsCount", a."DoubleBakingCount", a."DoubleEndorsingCount", a."EndorsementsCount", a."FrozenDeposits", a."FrozenFees", a."FrozenRewards", a."NonceRevelationsCount", a."ProposalsCount", a."RevelationPenaltiesCount", a."SoftwareId", a."StakingBalance"
      FROM "Accounts" AS a
      WHERE a."Address" = @__address_0
      LIMIT 1
fail: Microsoft.EntityFrameworkCore.Query[10100]
      An exception occurred while iterating over the results of a query for context type 'Tzkt.Data.TzktContext'.
      Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Copy'
         at Npgsql.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|233_0(<>c__DisplayClass233_0& )
         at Npgsql.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
      Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Copy'
         at Npgsql.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|233_0(<>c__DisplayClass233_0& )
         at Npgsql.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The connection is already in state 'Copy'
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (1ms) [Parameters=[@__address_0='?' (DbType = StringFixedLength)], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."Address", a."Balance", a."ContractsCount", a."Counter", a."DelegateId", a."DelegationLevel", a."DelegationsCount", a."FirstLevel", a."LastLevel", a."MigrationsCount", a."OriginationsCount", a."RevealsCount", a."Staked", a."TransactionsCount", a."Type", a."CreatorId", a."Kind", a."ManagerId", a."Spendable", a."Tzips", a."WeirdDelegateId", a."Activated", a."PublicKey", a."Revealed", a."ActivationLevel", a."BallotsCount", a."BlocksCount", a."DeactivationLevel", a."DelegatorsCount", a."DoubleBakingCount", a."DoubleEndorsingCount", a."EndorsementsCount", a."FrozenDeposits", a."FrozenFees", a."FrozenRewards", a."NonceRevelationsCount", a."ProposalsCount", a."RevelationPenaltiesCount", a."SoftwareId", a."StakingBalance"
      FROM "Accounts" AS a
      WHERE a."Address" = @__address_0
      LIMIT 1
fail: Microsoft.EntityFrameworkCore.Query[10100]
      An exception occurred while iterating over the results of a query for context type 'Tzkt.Data.TzktContext'.
      Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Copy'
         at Npgsql.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|233_0(<>c__DisplayClass233_0& )
         at Npgsql.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
      Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Copy'
         at Npgsql.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|233_0(<>c__DisplayClass233_0& )
         at Npgsql.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The connection is already in state 'Copy'
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (0ms) [Parameters=[@__address_0='?' (DbType = StringFixedLength)], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."Address", a."Balance", a."ContractsCount", a."Counter", a."DelegateId", a."DelegationLevel", a."DelegationsCount", a."FirstLevel", a."LastLevel", a."MigrationsCount", a."OriginationsCount", a."RevealsCount", a."Staked", a."TransactionsCount", a."Type", a."CreatorId", a."Kind", a."ManagerId", a."Spendable", a."Tzips", a."WeirdDelegateId", a."Activated", a."PublicKey", a."Revealed", a."ActivationLevel", a."BallotsCount", a."BlocksCount", a."DeactivationLevel", a."DelegatorsCount", a."DoubleBakingCount", a."DoubleEndorsingCount", a."EndorsementsCount", a."FrozenDeposits", a."FrozenFees", a."FrozenRewards", a."NonceRevelationsCount", a."ProposalsCount", a."RevelationPenaltiesCount", a."SoftwareId", a."StakingBalance"
      FROM "Accounts" AS a
      WHERE a."Address" = @__address_0
      LIMIT 1

Where are the RPC JSON parsing rules?

I would like to makes changes to implement and adapt this fantastic block explorer to a new testnet and which has modified RPC responses. Indeed, my interest is to make changes to the JSON parsing of the operation of Endorsement and include also a new type of operation. Which files should I focus on in the codebase ?

I had a look at Tzkt.Api/Models/Operations/EndorsementOperation.cs and Tzkt.Data/Models/Operations/EndorsementOperation.cs but I don't see where the parsing rules occur.

CSV exports for every table

It would be a neat feature if, every time you see a page with tabular data (e.g. contracts related to an address, operations with an account, etc), you had a download button to download the table as a csv file. The alternative is copy paste, but tzkt tables are not copy paste friendly.

I can't access tzkt explorer through api endpoint

I have run tzkt successfully the logs from tzkt-api is given below
info: Program[0]
Version 1.12.4.0
info: Program[0]
Initialize database...
warn: Program[0]
12 pending migrations. Let's wait for the indexer to migrate the database, and try again.
info: Program[0]
Initialize database...
warn: Program[0]
9 pending migrations. Let's wait for the indexer to migrate the database, and try again.
info: Program[0]
Initialize database...
warn: Program[0]
No data in the database. Let's wait for the indexer to index at least two blocks, and try again.
info: Program[0]
Initialize database...
warn: Program[0]
No data in the database. Let's wait for the indexer to index at least two blocks, and try again.
info: Program[0]
Initialize database...
info: Program[0]
Database initialized
info: Tzkt.Api.Services.Cache.StateCache[0]
Loaded state [2:BKnnrpuWjPMHczEadcztegzHbLRR4f8j7oarPaDvuw4sxVtLZJp]
info: Tzkt.Api.Services.Cache.AccountsCache[0]
Loaded 7 of 10 accounts
info: Tzkt.Api.Services.Cache.AliasesCache[0]
Loaded 0 aliases
info: Tzkt.Api.Services.Cache.ProtocolsCache[0]
Loaded 3 of 3 protocols
info: Tzkt.Api.Services.Cache.TimeCache[0]
Loaded 3 timestamps
info: Tzkt.Api.Services.Cache.QuotesCache[0]
Loaded 3 quotes
info: Tzkt.Api.Services.Sync.StateListener[0]
DB listener started
info: Tzkt.Api.Services.Sync.StateListener[0]
Db listener connected
warn: Microsoft.AspNetCore.Server.Kestrel[0]
Overriding address(es) 'http://+:5000'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://0.0.0.0:5000
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: /app

  But the default api is not available at http://127.0.0.1:5000/

Tezos Domains Integration

Hi Team,

I was wondering when and if you are doing a Tezos Domains integration. We are using it and I think it is sticking around long and successful enough to start an integration in the "search bar"?

Thnak you for your feedback!

Best regards
Carlo

Strange indexer sync behavior

I am see the output like this while running an indexer on ghostnet

Looks like syncing is still going on while it get 404 response from rpc endpoint ?

Please suggest service requirements

Please advise the recommended system requirements.
You already mentioned that it has low resource consumption (4GB of RAM)
Can you suggest the volume size required to fit the restore db snapshot (mainnet)?
What is more resource intensive part of the system? My guess it's a sync.

Duplicate and overlapping DB indexes

Hi, I noticed a few duplicate DB indexes and indexes overlapping each other.

Removing them would make writes (inserts/updates) slightly faster (doesn't matter that much for TzKT in my experience) and reduce the DB size without impact on read performances.

The two cases are:

More than one unique index on a same column (PK is always unique, having a two UNIQUE INDEX on the PK doesn't serve any purpose)
Overlapping index: INDEX ix_foo ON the_table(foo) when INDEX ix_foo_bar ON the_table(foo, bar) also exists
In this situation if we didn't have ix_foo, querying the_table with WHERE foo (and/or ORDER BY foo) would use ix_foo_bar.
It's not exactly as simple as that because ix_foo_bar is bigger than ix_foo and ix_foo can have benefits when postgres is combining indexes. I'd totally understand if you want to keep (some of?) those.
I know it's not easy to measure in production because in this situation we wouldn't see ix_foo as unused index. The performances shouldn't be much worse after removing it though, since pg would use ix_foo_bar instead of ix_foo.

On table: BakerCycles

--- overlap: when querying by `Cycle` only, `IX_BakerCycles_Cycle_BakerId` will be used
-CREATE INDEX "IX_BakerCycles_Cycle" ON "BakerCycles"("Cycle" int4_ops);
 CREATE UNIQUE INDEX "IX_BakerCycles_Cycle_BakerId" ON "BakerCycles"("Cycle" int4_ops,"BakerId" int4_ops);
--- duplicate
-CREATE UNIQUE INDEX "IX_BakerCycles_Id" ON "BakerCycles"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_BakerCycles" ON "BakerCycles"("Id" int4_ops);

On table: BakingRights

--- overlap: already covered by `IX_BakingRights_Cycle_BakerId`
-CREATE INDEX "IX_BakingRights_Cycle" ON "BakingRights"("Cycle" int4_ops);
 CREATE INDEX "IX_BakingRights_Cycle_BakerId" ON "BakingRights"("Cycle" int4_ops,"BakerId" int4_ops);

On table: BigMapKeys

--- overlap: already covered by `IX_BigMapKeys_BigMapPtr_KeyHash`
-CREATE INDEX "IX_BigMapKeys_BigMapPtr" ON "BigMapKeys"("BigMapPtr" int4_ops);
 CREATE INDEX "IX_BigMapKeys_BigMapPtr_KeyHash" ON "BigMapKeys"("BigMapPtr" int4_ops,"KeyHash" text_ops);

On table: BigMaps

--- duplicate
-CREATE UNIQUE INDEX "AK_BigMaps_Ptr" ON "BigMaps"("Ptr" int4_ops);
 CREATE UNIQUE INDEX "IX_BigMaps_Ptr" ON "BigMaps"("Ptr" int4_ops);
--- duplicate
-CREATE UNIQUE INDEX "IX_BigMaps_Id" ON "BigMaps"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_BigMaps" ON "BigMaps"("Id" int4_ops);

On table: BigMapUpdates

--- duplicate
-CREATE UNIQUE INDEX "IX_BigMapUpdates_Id" ON "BigMapUpdates"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_BigMapUpdates" ON "BigMapUpdates"("Id" int4_ops);

On table: Commitments

--- duplicate
-CREATE UNIQUE INDEX "IX_Commitments_Id" ON "Commitments"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_Commitments" ON "Commitments"("Id" int4_ops);

On table: Cycles

--- duplicate
-CREATE UNIQUE INDEX "AK_Cycles_Index" ON "Cycles"("Index" int4_ops);
 CREATE UNIQUE INDEX "IX_Cycles_Index" ON "Cycles"("Index" int4_ops);

On table: DelegatorCycles

--- overlap
-CREATE INDEX "IX_DelegatorCycles_Cycle" ON "DelegatorCycles"("Cycle" int4_ops);
 CREATE INDEX "IX_DelegatorCycles_Cycle_BakerId" ON "DelegatorCycles"("Cycle" int4_ops,"BakerId" int4_ops);

On table: Events

--- overlap
-CREATE INDEX "IX_Events_ContractCodeHash" ON "Events"("ContractCodeHash" int4_ops);
 CREATE INDEX "IX_Events_ContractCodeHash_Tag" ON "Events"("ContractCodeHash" int4_ops,"Tag" text_ops);
--- overlap
-CREATE INDEX "IX_Events_ContractId" ON "Events"("ContractId" int4_ops);
 CREATE INDEX "IX_Events_ContractId_Tag" ON "Events"("ContractId" int4_ops,"Tag" text_ops);
--- duplicate
-CREATE UNIQUE INDEX "IX_Events_Id" ON "Events"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_Events" ON "Events"("Id" int4_ops);

On table: Scripts

--- duplicate
-CREATE UNIQUE INDEX "IX_Scripts_Id" ON "Scripts"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_Scripts" ON "Scripts"("Id" int4_ops);

On table: SmartRollupCommitments

--- overlap
-CREATE INDEX "IX_SmartRollupCommitments_Hash" ON "SmartRollupCommitments"("Hash" text_ops);
 CREATE INDEX "IX_SmartRollupCommitments_Hash_SmartRollupId" ON "SmartRollupCommitments"("Hash" text_ops,"SmartRollupId" int4_ops);

On table: Storages

--- duplicate
-CREATE UNIQUE INDEX "IX_Storages_Id" ON "Storages"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_Storages" ON "Storages"("Id" int4_ops);

On table: TokenBalances

--- duplicate
-CREATE UNIQUE INDEX "IX_TokenBalances_Id" ON "TokenBalances"("Id" int8_ops);
 CREATE UNIQUE INDEX "PK_TokenBalances" ON "TokenBalances"("Id" int8_ops);

On table: Tokens

--- overlap
-CREATE INDEX "IX_Tokens_ContractId" ON "Tokens"("ContractId" int4_ops);
 CREATE UNIQUE INDEX "IX_Tokens_ContractId_TokenId" ON "Tokens"("ContractId" int4_ops,"TokenId" text_ops);
--- duplicate
-CREATE UNIQUE INDEX "IX_Tokens_Id" ON "Tokens"("Id" int8_ops);
 CREATE UNIQUE INDEX "PK_Tokens" ON "Tokens"("Id" int8_ops);

On table: TokenTransfers

--- duplicate
-CREATE UNIQUE INDEX "IX_TokenTransfers_Id" ON "TokenTransfers"("Id" int8_ops);
 CREATE UNIQUE INDEX "PK_TokenTransfers" ON "TokenTransfers"("Id" int8_ops);

On table: VotingPeriods

--- duplicate
-CREATE UNIQUE INDEX "AK_VotingPeriods_Index" ON "VotingPeriods"("Index" int4_ops);
 CREATE UNIQUE INDEX "IX_VotingPeriods_Index" ON "VotingPeriods"("Index" int4_ops);
--- duplicate
-CREATE UNIQUE INDEX "IX_VotingPeriods_Id" ON "VotingPeriods"("Id" int4_ops);
 CREATE UNIQUE INDEX "PK_VotingPeriods" ON "VotingPeriods"("Id" int4_ops);

On table: VotingSnapshots

--- overlap
-CREATE INDEX "IX_VotingSnapshots_Period" ON "VotingSnapshots"("Period" int4_ops);
 CREATE UNIQUE INDEX "IX_VotingSnapshots_Period_BakerId" ON "VotingSnapshots"("Period" int4_ops,"BakerId" int4_ops);

I guess my main suggestion is to first drop all duplicate indexes:

DROP INDEX public."IX_BakerCycles_Id";
DROP INDEX public."AK_BigMaps_Ptr";
DROP INDEX public."IX_BigMaps_Id";
DROP INDEX public."IX_BigMapUpdates_Id";
DROP INDEX public."IX_Commitments_Id";
DROP INDEX public."AK_Cycles_Index";
DROP INDEX public."IX_Events_Id";
DROP INDEX public."IX_Scripts_Id";
DROP INDEX public."IX_Storages_Id";
DROP INDEX public."IX_TokenBalances_Id";
DROP INDEX public."IX_Tokens_Id";
DROP INDEX public."IX_TokenTransfers_Id";
DROP INDEX public."AK_VotingPeriods_Index";
DROP INDEX public."IX_VotingPeriods_Id";

and carefully reconsider the overlapping ones.

Getting Storage Diff on Operation Level

I am wondering if it's possible to get storage diff from /v1/operations/transactions similar to the result of https://api.better-call.dev/v1/opg/xxx

Tzkt.Sync no longer working on ghostnet

Hello team,
I'm running completely identical configurations of tzkt and octez for mainnet and ghostnet. However since 2023-10-21 Tzkt.Sync stopped working on ghostnet. I've tried to download a fresh tzkt database snapshot and restarted all container,but still receiving the same error. Octez ghostnet is up and synced.

tzkt-sync: v1.12.4
tzkt-api: v1.12.4
octez: v18.0

Error logs:

2023-10-25T15:16:27.542397182Z info: Tzkt.Sync.Services.Observer[0]
2023-10-25T15:16:27.542407182Z       Synchronization started
2023-10-25T15:16:29.132813228Z fail: Tzkt.Sync.Services.Observer[0]
2023-10-25T15:16:29.132854758Z       Failed to apply updates
2023-10-25T15:16:29.132863489Z       System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
2023-10-25T15:16:29.132871309Z          at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
2023-10-25T15:16:29.132878699Z          at System.Net.Http.HttpClient.GetStreamAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
2023-10-25T15:16:29.132885669Z          at Tzkt.Sync.Services.TezosNode.GetAsync(String url) in /app/Tzkt.Sync/Services/TezosNode/TezosNode.cs:line 32
2023-10-25T15:16:29.132892809Z          at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 52
2023-10-25T15:16:29.132899109Z          at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 183
2023-10-25T15:16:29.132904299Z          at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 66
2023-10-25T15:16:32.153082197Z fail: Tzkt.Sync.Services.Observer[0]
2023-10-25T15:16:32.153104387Z       Failed to apply updates
2023-10-25T15:16:32.153111617Z       System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
2023-10-25T15:16:32.153118307Z          at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
2023-10-25T15:16:32.153124347Z          at System.Net.Http.HttpClient.GetStreamAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
2023-10-25T15:16:32.153131247Z          at Tzkt.Sync.Services.TezosNode.GetAsync(String url) in /app/Tzkt.Sync/Services/TezosNode/TezosNode.cs:line 32
2023-10-25T15:16:32.153138937Z          at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 52
2023-10-25T15:16:32.153158498Z          at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 183
2023-10-25T15:16:32.153164918Z          at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 66
2023-10-25T15:16:35.224714450Z fail: Tzkt.Sync.Services.Observer[0]
2023-10-25T15:16:35.224770341Z       Failed to apply updates
2023-10-25T15:16:35.224787171Z       System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
2023-10-25T15:16:35.224799601Z          at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
2023-10-25T15:16:35.224809891Z          at System.Net.Http.HttpClient.GetStreamAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
2023-10-25T15:16:35.224820131Z          at Tzkt.Sync.Services.TezosNode.GetAsync(String url) in /app/Tzkt.Sync/Services/TezosNode/TezosNode.cs:line 32
2023-10-25T15:16:35.224831031Z          at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 52
2023-10-25T15:16:35.224844132Z          at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 183
2023-10-25T15:16:35.224857552Z          at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 66

Edit: I'm running octez with historyMode rolling, I'll increase the cycle and try again - I think this might fix it

baking-bad / tzkt Goto Github PK

tzkt's People

Stargazers

Watchers

Forkers

tzkt's Issues

Recommend Projects

Recommend Topics

Recommend Org