Comments (17)
dgaubert
commented on July 18, 2024
1
Back from vacations!
please let @CartoDB/rt-managers now if this is appropriate to tackle at the RT
Sure thing!
from camshaft.
dgaubert
commented on July 18, 2024
1
@shivam8800 are you setting up a log rotator in your installation?. Or are you experiencing the same issue with Bunyan?
This issue is related to the integration among Windshaft-CartoDB, Bunyan and the log rotator. For Bunyan's support, please, open an issue in Bunyan's repository.
from camshaft.
ramiroaznar
commented on July 18, 2024
@Algunenano could you share more info? Backend/Infra? What is the priority of this issue?
from camshaft.
Algunenano
commented on July 18, 2024
Backend/Infra?
Backend. It's node code.
What is the priority of this issue?
The process kept running, but it's left in an unknown status. In fact, more modern stable releases of node would have killed the process. So it's not high priority but something we should look into sooner rather than later.
from camshaft.
Algunenano
commented on July 18, 2024
As a side note, if CartoDB/Windshaft-cartodb#1065 was already place the process would have shutdown and restarted, and in this case it should be avoided.
from camshaft.
dgaubert
commented on July 18, 2024
This is happening, under some circumstances, while rotating the logging file for analysis: node-windshaft-analysis.log (once per day, usually at 00:17 AM). The stream to that file has been closed and Bunyan is not able to refresh the file descriptor before writing a new entry log despite we've implemented a callback to refresh it. We had a similar issue while implementing cluster-mode (plus graceful shutdown) for Tilesets API and I had to overwrite the method .close() in Logger to handle it properly.
We need to spend some time to determine the right place and approach to fix it.
from camshaft.
jsanz
commented on July 18, 2024
@dgaubert please let @CartoDB/rt-managers now if this is appropriate to tackle at the RT
from camshaft.
dgaubert
commented on July 18, 2024
Fixed: #384
from camshaft.
Algunenano
commented on July 18, 2024
It happened again tonight:
[2019-06-04 00:17:26.061] [FATAL] [default] - Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at WriteStream.Writable.write (_stream_writable.js:294:11)
at Logger._emit (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/bunyan/lib/bunyan.js:938:22)
at Logger.info (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/bunyan/lib/bunyan.js:1037:24)
at AnalysisLogger.logLimitsError (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/logging/logger.js:41:21)
at LimitsContext.logError (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/limits/context.js:18:21)
at Object.limitNumberOfRows (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/node/limits.js:10:17)
at BaseNode.Node.checkLimits (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/node/node.js:535:12)
at /home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/limits/checker.js:12:17
Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at WriteStream.Writable.write (_stream_writable.js:294:11)
at Logger._emit (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/bunyan/lib/bunyan.js:938:22)
at Logger.info (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/bunyan/lib/bunyan.js:1037:24)
at AnalysisLogger.logLimitsError (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/logging/logger.js:41:21)
at LimitsContext.logError (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/limits/context.js:18:21)
at Object.limitNumberOfRows (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/node/limits.js:10:17)
at BaseNode.Node.checkLimits (/home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/node/node.js:535:12)
at /home/ubuntu/www/node-windshaft/releases/20190521103931/node_modules/camshaft/lib/limits/checker.js:12:17
[2019-06-04 00:17:26.062] [INFO] [default] - Process is going to exit with code: 1
[2019-06-04 00:18:11.063] [INFO] [default] - Process didn't close on time. Force exit
[2019-06-04 00:18:15.816] [INFO] console - {"name":"windshaft-server","db":8,"action":"acquire","elapsed":103,"waiting":0}
Notice it's a fatal error so it's restarting the service.
from camshaft.
dgaubert
commented on July 18, 2024
Hey!
Since we implemented graceful shutdown, we are not leaving any request w/o response. In any case, we should fix it. This issue fits to be tackled in RT.
cc/ @CartoDB/rt-managers
from camshaft.
Algunenano
commented on July 18, 2024
And again:
[2019-07-28 00:17:36.031] [FATAL] [default] - Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at WriteStream.Writable.write (_stream_writable.js:294:11)
at Logger._emit (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/bunyan/lib/bunyan.js:938:22)
at Logger.info (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/bunyan/lib/bunyan.js:1037:24)
at AnalysisLogger.logLimitsError (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/logging/logger.js:41:21)
at LimitsContext.logError (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/limits/context.js:18:21)
at Object.limitNumberOfRows (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/node/limits.js:10:17)
at BaseNode.Node.checkLimits (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/node/node.js:535:12)
at /home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/limits/checker.js:12:17
Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at WriteStream.Writable.write (_stream_writable.js:294:11)
at Logger._emit (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/bunyan/lib/bunyan.js:938:22)
at Logger.info (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/bunyan/lib/bunyan.js:1037:24)
at AnalysisLogger.logLimitsError (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/logging/logger.js:41:21)
at LimitsContext.logError (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/limits/context.js:18:21)
at Object.limitNumberOfRows (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/node/limits.js:10:17)
at BaseNode.Node.checkLimits (/home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/node/node.js:535:12)
at /home/ubuntu/www/node-windshaft/releases/20190723161603/node_modules/camshaft/lib/limits/checker.js:12:17
[2019-07-28 00:17:36.032] [INFO] [default] - Process is going to exit with code: 1
[2019-07-28 00:18:21.033] [INFO] [default] - Process didn't close on time. Force exit
from camshaft.
Algunenano
commented on July 18, 2024
Again:
[2019-08-12 00:17:33.055] [FATAL] [default] - Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at WriteStream.Writable.write (_stream_writable.js:294:11)
at Logger._emit (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/bunyan/lib/bunyan.js:938:22)
at Logger.info (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/bunyan/lib/bunyan.js:1037:24)
at AnalysisLogger.logLimitsError (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/logging/logger.js:41:21)
at LimitsContext.logError (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/limits/context.js:18:21)
at Object.limitNumberOfRows (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/node/limits.js:10:17)
at BaseNode.Node.checkLimits (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/node/node.js:535:12)
at /home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/limits/checker.js:12:17
Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at WriteStream.Writable.write (_stream_writable.js:294:11)
at Logger._emit (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/bunyan/lib/bunyan.js:938:22)
at Logger.info (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/bunyan/lib/bunyan.js:1037:24)
at AnalysisLogger.logLimitsError (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/logging/logger.js:41:21)
at LimitsContext.logError (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/limits/context.js:18:21)
at Object.limitNumberOfRows (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/node/limits.js:10:17)
at BaseNode.Node.checkLimits (/home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/node/node.js:535:12)
at /home/ubuntu/www/node-windshaft/releases/20190730102543/node_modules/camshaft/lib/limits/checker.js:12:17
[2019-08-12 00:17:33.056] [INFO] [default] - Process is going to exit with code: 1
[2019-08-12 00:18:18.057] [INFO] [default] - Process didn't close on time. Force exit
[2019-08-12 00:18:49.980] [INFO] console - {"name":"windshaft-server","db":8,"action":"acquire","elapsed":1017,"waiting":0}
from camshaft.
shivam8800
commented on July 18, 2024
is there any solution for this error ?
I am stuck in it and don't know what to do
from camshaft.
shivam8800
commented on July 18, 2024
Oh got it,
Actually, It was from pm2
Sorry for mentioning here :(
from camshaft.
nickgrealy
commented on July 18, 2024
I got this error... searched further up in the logs and found this message, which was the cause:
Unhandled rejection Error: ENOSPC: no space left on device, write
from camshaft.
Jarryxin
commented on July 18, 2024
@shivam8800 are you setting up a log rotator in your installation?. Or are you experiencing the same issue with Bunyan?
This issue is related to the integration among
Windshaft-CartoDB,Bunyanand thelog rotator. For Bunyan's support, please, open an issue in Bunyan's repository.
确实由 Bunyan 引起的 😂
from camshaft.
Algunenano
commented on July 18, 2024
Closing stale ticket
from camshaft.
Related Issues (20)
- DO: Error augmenting with text columns HOT 1
- Allow executing arbitrary SQL for analysis preparation HOT 7
- Performance research: run independent analyses in parallel HOT 1
- Prevent column name collisions in node queries HOT 2
- Depricated SQL nodes not working HOT 18
- Review sql queries to improve planning with IMMUTABLE and PARALLEL functions HOT 6
- Invalid geometries generated with Area of Intereset HOT 1
- DO analysis fails with multiple columns with the same numerator HOT 2
- Poor SQL for nearest neighbor join
- Force "merge" analysis to be cached HOT 2
- Map instantiation: Get columns from cache table instead of using `LIMIT 0` when avaiblable HOT 9
- Add .npmignore HOT 1
- IMHO Misleading `limits_error` message
- Uncaught exception in some Batch API errors HOT 1
- Polygons from geometries analysis should be cached HOT 1
- Error with reserved words
- Replace CDB_QueryTables_Updated_At with cartodb-query-tables HOT 1
- Noise in the analysis logs HOT 4
- Uncaught exception: Range filter expect to have at least one value in greater_than, greater_than_or_equal, less_than, less_than_or_equal, min, or max numeric params
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from camshaft.