Comments (3)
I happened to just now notice some slow performance when accessing that Shasta S3 bucket. On closer inspection it appears compaction has not run successfully on the Autoperf pool since April, even though the zed manage
process has been running this whole time. That makes me think that the extra-slow performance puling so many tiny objects may go a long way toward aggravating this problem. If we end up fixing whatever's wrong there such that it compacts again, this crash might magically go away, but if so I could probably create the repro conditions again by manually creating a fragmented, non-compacted pool.
from zed.
tl;dr
I've attached a shell script repro.sh.gz that reliably triggers the repro for me with GA Zed tagged v1.17.0
.
Before running, start:
zed -lake s3://brimdataio-lakes-us-east-2/shasta.lake.brimdata.io serve
Details
I made this script by using Wireshark to sniff the queries to the Zed lake API that were being issued by Grafana when loading the dashboard. There's actually more queries in the pcap than I transcribed (i.e., I only made it as big as I needed to trigger the problem) so if this doesn't work for others on the first try (geographic locations that see lower latency to the S3 bucket in Ohio might fare better than me) just say the word and I can add more.
from zed.
Per the previous comment, the Zed services on Shasta have been updated to GA v1.17.0
, compaction ran successfully, and indeed the problem no longer reproduces when the repro script is run against the pool data in this compacted form. However, anticipating this, I saved off the pool beforehand in its fragmented state, so to repro the problem, now do:
zed -lake s3://brimdataio-lakes-us-east-2/shasta-fragmented/shasta.lake.brimdata.io serve
...and the repro script still triggers the crash just as before. I just confirmed this with Zed commit 54a5a15.
from zed.
Related Issues (20)
- Can't successfully pass "this" as part of an array parameter to user-defined op HOT 2
- Something like jq's "debug" HOT 2
- Accessing the index of current array element within an "over" scope HOT 1
- "Strict mode" for Zed language
- search_replace function HOT 2
- Spread values from an array to become the multiple args to coalesce()
- Show filename when searching across multiple files with zq
- "this" as optional second argument in functions
- Dynamic field names in record expressions HOT 1
- Incorrect comparison result with union value HOT 1
- Most functions don't support unions
- Warn/error on "over ... with this"
- [SUGESTION] Syntax highlighting for HTML template literals HOT 1
- Nanosecond precision output with strftime
- Presenting times in non-UTC timezones
- Hung debug with larger input and head HOT 1
- Post-merge operator prevents debug on flowgraph branches
- "sort" causes Zed lake service crash
- Don't colorize redirected debug output
- nest_dotted function panics on non-record value
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zed.