Comments (10)
evoskuil
commented on July 19, 2024
This has also been observed on stealth_rows:
09:30:08.175011 DEBUG [network] Manual channel stopped: success
09:30:08.175011 DEBUG [network] Suspended manual connection.
09:30:08.175011 DEBUG [network] Stopped block_in protocol for [139.59.210.142:8333].
09:30:08.175011 DEBUG [network] Stopped block_out protocol for [139.59.210.142:8333].
09:30:08.175011 DEBUG [network] Stopped transaction_in protocol for [139.59.210.142:8333].
09:30:08.175011 DEBUG [network] Stopped transaction_out protocol for [139.59.210.142:8333].
09:30:08.221915 DEBUG [database] Unmapped: "d:\server\blockchain\block_table" [6299560045]
09:30:08.221915 DEBUG [database] Unmapped: "d:\server\blockchain\block_index" [3617948]
09:30:11.925169 DEBUG [database] Unmapped: "d:\server\blockchain\transaction_table" [115081554420]
09:30:13.263837 DEBUG [database] Unmapped: "d:\server\blockchain\spend_table" [1717490472]
09:30:14.381619 DEBUG [database] Unmapped: "d:\server\blockchain\history_table" [603877976]
09:30:14.397267 DEBUG [database] Unmapped: "d:\server\blockchain\history_rows" [1689414019]
09:30:14.397267 DEBUG [database] Unmapped: "d:\server\blockchain\stealth_rows" [246196]
09:30:14.397267 INFO [node] Server stopped successfully.
09:30:48.784253 INFO [server] Please wait while the server is starting...
09:30:48.784253 DEBUG [database] Buckets: block [650000], transaction [110000000], spend [250000000], history [107000000]
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\block_table" [6299560045] (4096)
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\block_index" [3617948] (4096)
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\transaction_table" [115081554420] (4096)
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\spend_table" [1717490472] (4096)
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\history_table" [603877976] (4096)
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\history_rows" [1689414019] (4096)
09:30:48.852814 DEBUG [database] Mapping: "d:\server\blockchain\stealth_rows" [246195] (4096)
09:30:48.852814 ERROR [node] Failure starting blockchain.
246195 vs. 246196 (also 1 byte short).
from libbitcoin-database.
evoskuil
commented on July 19, 2024
Based on the assumption that this is a bug in the Windows memory mapped file implementation we can save an extra arbitrary byte, which will be ignored if not truncated.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
This has not bee reproduced in over two months of continuous and dynamic operations on Windows. At this point we can reasonably assume it was a side effect of another issue that has been resolved.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
Still here:
06:27:08.196804 DEBUG [database] Unmapped: "d:\server\blockchain\block_table" [2653848983]
06:27:08.370930 DEBUG [database] Unmapped: "d:\server\blockchain\block_index" [2973516]
06:28:56.858253 DEBUG [database] Unmapped: "d:\server\blockchain\transaction_table" [48257495656]
06:29:18.075754 DEBUG [database] Unmapped: "d:\server\blockchain\spend_table" [17000507932]
06:29:28.993141 DEBUG [database] Unmapped: "d:\server\blockchain\history_table" [3000068920]
06:29:34.625789 DEBUG [database] Unmapped: "d:\server\blockchain\history_rows" [38309327284]
06:29:34.801125 DEBUG [database] Unmapped: "d:\server\blockchain\stealth_rows" [854224]
06:29:34.801125 INFO [node] Server stopped successfully.
06:30:36.970219 DEBUG [database] Mapping: "d:\server\blockchain\block_table" [2653848983] (4096)
06:30:36.970219 DEBUG [database] Mapping: "d:\server\blockchain\block_index" [2973516] (4096)
06:30:36.970219 DEBUG [database] Mapping: "d:\server\blockchain\transaction_table" [48257495656] (4096)
06:30:36.971219 DEBUG [database] Mapping: "d:\server\blockchain\spend_table" [17000507932] (4096)
06:30:36.971219 DEBUG [database] Mapping: "d:\server\blockchain\history_table" [3000068919] (4096)
06:30:36.971219 DEBUG [database] Mapping: "d:\server\blockchain\history_rows" [38309327284] (4096)
06:30:36.971219 ERROR [node] Failure starting blockchain.
3000068919 vs. 3000068920 (also 1 byte short).
from libbitcoin-database.
evoskuil
commented on July 19, 2024
The problem above is recorded on three different files, so clearly this is related to the underlying memory mapping: memory_map.cpp, mman.c or the Windows API.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
After adding logging and additional error handling... Both show that the physical file size is larger than the logical size at the time of truncation, and that the truncation size specified is the correct logical size (this has been verified in the debugger).
23:35:41.011502 DEBUG [database] Unmapped: "d:\server\blockchain\block_table" [2612808684, 3285300237]
23:35:41.110902 DEBUG [database] Unmapped: "d:\server\blockchain\block_index" [2960004, 4247430]
23:36:55.356525 DEBUG [database] Unmapped: "d:\server\blockchain\transaction_table" [47496894744, 58548700387]
23:37:05.313473 DEBUG [database] Unmapped: "d:\server\blockchain\spend_table" [16732364276, 20802165714]
23:37:16.746676 DEBUG [database] Unmapped: "d:\server\blockchain\history_table" [2952145856, 3742726242]
23:37:40.990499 DEBUG [database] Unmapped: "d:\server\blockchain\history_rows" [37701078079, 45329657188]
23:37:40.995745 DEBUG [database] Unmapped: "d:\server\blockchain\stealth_rows" [792492, 879204]
23:37:40.995745 INFO [node] Server stopped successfully.
23:46:18.676643 DEBUG [database] Mapping: "d:\server\blockchain\block_table" [2612808684] (4096)
23:46:18.676643 DEBUG [database] Mapping: "d:\server\blockchain\block_index" [2960004] (4096)
23:46:18.677672 DEBUG [database] Mapping: "d:\server\blockchain\transaction_table" [47496894744] (4096)
23:46:18.677672 DEBUG [database] Mapping: "d:\server\blockchain\spend_table" [16732364276] (4096)
23:46:18.677672 DEBUG [database] Mapping: "d:\server\blockchain\history_table" [2952145855] (4096)
23:46:18.687683 DEBUG [database] Mapping: "d:\server\blockchain\history_rows" [37701078079] (4096)
23:46:18.687683 ERROR [node] Failure starting blockchain.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
In the above example, the file system reports the file size after the store close as 2,952,145,856 bytes, which is the size reportedly written and which is expected at open (via the debugger).
After failing to open the store the implementation closes it, resulting in the one byte reduction of its size, matching the size reported in that log for both open and close.
In other words, the file is correctly sized on disk but the opening of the file fails to register the correct size, with the resulting behavior being expected.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
By taking a snapshot of the store before attempting to reopen, we confirmed the above and were able to retry opening copies of the same store multiple times. The first attempt failed (above), and it was just coincidental that we had the snapshot. Subsequent attempts from the snapshot also failed. So even though the file is reported with the correct size, when the operating system opens the file the size it reports is less one byte.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
The debugger confirms that _fstat64(...) returns a structure with the incorrect size for the file (one byte less than written and than what the Windows Explorer reports).
04/12/2017 05:49 PM <DIR> .
04/12/2017 05:49 PM <DIR> ..
04/12/2017 05:18 PM 2,960,004 block_index
04/12/2017 05:18 PM 2,612,808,684 block_table
04/12/2017 05:18 PM 37,701,078,079 history_rows
04/12/2017 04:37 PM 2,952,145,856 history_table <=====
04/12/2017 05:18 PM 16,732,364,276 spend_table
04/12/2017 04:37 PM 792,492 stealth_rows
04/12/2017 05:18 PM 47,496,894,744 transaction_table
7 File(s) 107,499,044,135 bytes
2 Dir(s) 84,900,995,072 bytes free
Simply opening the file reduces its reported size by the one byte.
from libbitcoin-database.
evoskuil
commented on July 19, 2024
Turns out that _wsopen() requires a special flag to prevent it from mutating the file. This problem is resolved by adding the _O_BINARY flag.
from libbitcoin-database.
Related Issues (20)
- Flush lock slows block out protocol noticeably. HOT 1
- Set file growth rate automatically if configured (default). HOT 1
- array_index is 32 bit, will require expansion. HOT 3
- call end_write before return HOT 1
- History store row allocator requires concurrency guard. HOT 14
- Rename history database to address database. HOT 1
- Hash tables not safe for read while conflict delete.
- Port resolution to issues #150 and #159 to master. HOT 1
- Changes to std::hash template for db keys limits portability. HOT 1
- Primitives require file_offset and array_index type parameterization. HOT 1
- Store tx offset vs. point in address row file. HOT 2
- [master] Header storage exceedingly slow. HOT 2
- Use tx link instead of tx hash for input_point. HOT 2
- Support unconfirmed tx as output spender. HOT 2
- [master] flush_lock must be file-specific for parallel write flushing. HOT 2
- Remove hash from block_database::unindex or check that hash matches HOT 2
- What is the expected behavior for hash_table_multimap::find(Link link) behaviour with non existing link argument? HOT 7
- Add configurable operational file size minimums. HOT 1
- [master] Build warnings. HOT 3
- MacOS (all): no template named 'unary_function' in namespace 'std'.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libbitcoin-database.