Comments (10)
camertron
commented on August 27, 2024
@rept hmm I'm not 100% sure what's going on without a little more information, but it sounds like you might not have a 500.html file in your public/ directory? The readiness probe in the assets pod expects 500.html to exist. Can you confirm 500.html exists in your project?
from kuby-core.
rept
commented on August 27, 2024
Yes it does, it's a brand new completely blank rails app. When running locally and going to /500.html it works fine.
DB is also causing issues, but I first thought this was related.
This is the complete log.
[FATAL][2021-04-16 22:41:25 +0200] Deployment/lennis-assets: TIMED OUT (progress deadline: 600s)
[FATAL][2021-04-16 22:41:25 +0200] Timeout reason: ProgressDeadlineExceeded
[FATAL][2021-04-16 22:41:25 +0200] Latest ReplicaSet: lennis-assets-8b866d58
[FATAL][2021-04-16 22:41:25 +0200]
[FATAL][2021-04-16 22:41:25 +0200] The following containers have not passed their readiness probes on at least one pod:
[FATAL][2021-04-16 22:41:25 +0200] > lennis-assets must respond with a good status code at '/500.html'
[FATAL][2021-04-16 22:41:25 +0200]
[FATAL][2021-04-16 22:41:25 +0200] - Final status: 1 replica, 1 updatedReplica, 1 unavailableReplica
[FATAL][2021-04-16 22:41:25 +0200] - Events: None found. Please check your usual logging service (e.g. Splunk).
[FATAL][2021-04-16 22:41:25 +0200] - Logs from container 'lennis-assets' (last 25 lines shown):
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:40:49 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:40:52 [error] 6#6: *56289 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:40:52 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:40:55 [error] 6#6: *56290 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:40:55 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:40:58 [error] 6#6: *56291 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:40:58 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:01 [error] 6#6: *56292 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:01 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:04 [error] 6#6: *56293 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:04 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:07 [error] 6#6: *56294 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:07 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:10 [error] 6#6: *56295 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:10 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:13 [error] 6#6: *56296 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:13 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:16 [error] 6#6: *56297 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:16 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:19 [error] 6#6: *56298 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:19 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:22 [error] 6#6: *56299 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:22 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200] 2021/04/16 20:41:25 [error] 6#6: *56300 open() "/usr/share/nginx/assets/current/500.html" failed (2: No such file or directory), client: 10.244.0.76, server: localhost, request: "GET /500.html HTTP/1.1", host: "10.244.0.55:8082"
[FATAL][2021-04-16 22:41:25 +0200] 10.244.0.76 - - [16/Apr/2021:20:41:25 +0000] "GET /500.html HTTP/1.1" 404 169 "-" "kube-probe/1.20" "-"
[FATAL][2021-04-16 22:41:25 +0200]
[FATAL][2021-04-16 22:41:25 +0200] Deployment/lennis-web: FAILED
[FATAL][2021-04-16 22:41:25 +0200] Latest ReplicaSet: lennis-web-55bd976646
[FATAL][2021-04-16 22:41:25 +0200]
[FATAL][2021-04-16 22:41:25 +0200] The following containers are in a state that is unlikely to be recoverable:
[FATAL][2021-04-16 22:41:25 +0200] > lennis-create-db: Crashing repeatedly (exit 1). See logs for more information.
[FATAL][2021-04-16 22:41:25 +0200]
[FATAL][2021-04-16 22:41:25 +0200] - Final status: 1 replica, 1 updatedReplica, 1 unavailableReplica
[FATAL][2021-04-16 22:41:25 +0200] - Events (common success events excluded):
[FATAL][2021-04-16 22:41:25 +0200] [Pod/lennis-web-55bd976646-cbnrc] BackOff: Back-off restarting failed container (12896 events)
[FATAL][2021-04-16 22:41:25 +0200] - Logs from container 'lennis-web': None found. Please check your usual logging service (e.g. Splunk).
[FATAL][2021-04-16 22:41:25 +0200] - Logs from container 'lennis-migrate-db': None found. Please check your usual logging service (e.g. Splunk).
[FATAL][2021-04-16 22:41:25 +0200] - Logs from container 'lennis-copy-assets': None found. Please check your usual logging service (e.g. Splunk).
[FATAL][2021-04-16 22:41:25 +0200] - Logs from container 'lennis-create-db' (last 25 lines shown):
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/activerecord-6.1.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:866:in `acquire_connection'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/activerecord-6.1.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:588:in `checkout'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/activerecord-6.1.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:428:in `connection'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/activerecord-6.1.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1128:in `retrieve_connection'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/activerecord-6.1.3.1/lib/active_record/connection_handling.rb:327:in `retrieve_connection'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/activerecord-6.1.3.1/lib/active_record/connection_handling.rb:283:in `connection'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/kuby-core-0.11.14/lib/kuby/plugins/rails_app/tasks.rake:21:in `block (4 levels) in <main>'
[FATAL][2021-04-16 22:41:25 +0200] /usr/src/app/bundle/ruby/3.0.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63:in `load'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:63:in `kernel_load'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/cli/exec.rb:28:in `run'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/cli.rb:494:in `exec'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/cli.rb:30:in `dispatch'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/cli.rb:24:in `start'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/exe/bundle:49:in `block in <top (required)>'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/lib/bundler/friendly_errors.rb:130:in `with_friendly_errors'
[FATAL][2021-04-16 22:41:25 +0200] /usr/local/bundle/gems/bundler-2.2.15/exe/bundle:37:in `<top (required)>'
[FATAL][2021-04-16 22:41:25 +0200] bin/bundle:113:in `load'
[FATAL][2021-04-16 22:41:25 +0200] bin/bundle:113:in `<main>'
[FATAL][2021-04-16 22:41:25 +0200] Tasks: TOP => kuby:rails_app:db:create_unless_exists
[FATAL][2021-04-16 22:41:25 +0200] (See full trace by running task with --trace)
from kuby-core.
camertron
commented on August 27, 2024
Great, thanks for all the log output :) Hmm I wonder if this has something to do with changes in Rails 6.1. It does look like the readiness probe is getting a 404 when trying to retrieve 500.html. Let me spin up a new 6.1 app and see if I can reproduce the issue. In any case, I'm fairly certain the problem isn't related to the .app TLD.
from kuby-core.
camertron
commented on August 27, 2024
Hey @rept, I haven't been able to reproduce the problem. Can you copy/paste your kuby.rb file (passwords omitted, of course)?
from kuby-core.
rept
commented on August 27, 2024
@camertron of course.
require 'active_support/core_ext'
require 'active_support/encrypted_configuration'
require 'kuby/digitalocean'
# Define a production Kuby deploy environment
Kuby.define('lennis') do
environment(:production) do
# Because the Rails environment isn't always loaded when
# your Kuby config is loaded, provide access to Rails
# credentials manually.
app_creds = ActiveSupport::EncryptedConfiguration.new(
config_path: File.join('config', 'credentials.yml.enc'),
key_path: File.join('config', 'master.key'),
env_key: 'RAILS_MASTER_KEY',
raise_if_missing_key: true
)
docker do
# Configure your Docker registry credentials here. Add them to your
# Rails credentials file by running `bundle exec rake credentials:edit`.
credentials do
username app_creds[:KUBY_DOCKER_USERNAME]
password app_creds[:KUBY_DOCKER_PASSWORD]
email app_creds[:KUBY_DOCKER_EMAIL]
end
# Configure the URL to your Docker image here, eg:
# image_url 'foo.bar.com/me/myproject'
#
# If you're using Gitlab's Docker registry, try something like this:
image_url 'registry.gitlab.com/rept/XXXXX'
end
kubernetes do
# Add a plugin that facilitates deploying a Rails app.
add_plugin :rails_app do
hostname 'xxxxx.app'
# configure database credentials
database do
user 'xxxxx'
password app_creds[:DATABASE_PASSWORD]
end
end
# Use Docker Desktop as the provider.
# See: https://www.docker.com/products/docker-desktop
#
# Note: you will likely want to use a different provider when deploying
# your application into a production environment. To configure a different
# provider, add the corresponding gem to your gemfile and update the
# following line according to the provider gem's README.
provider :digitalocean do
access_token app_creds[:DIGITALOCEAN_ACCESS_TOKEN]
cluster_id app_creds[:DIGITALOCEAN_CLUSTER_ID]
end
end
end
end
from kuby-core.
rept
commented on August 27, 2024
Maybe also useful: database.yml
default: &default
adapter: postgresql
encoding: unicode
# For details on connection pooling, see Rails configuration guide
# https://guides.rubyonrails.org/configuring.html#database-pooling
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
development:
<<: *default
database: xxxxx_development
test:
<<: *default
database: xxxxx_test
production:
<<: *default
database: xxxxx_production
username: xxxxx
password: <%= Rails.application.credentials.dig(:DATABASE_PASSWORD) %>
The encrypted file looks like this:
secret_key_base: fjeffefejoifejoijoefizjfioezjofzejofiezjifpjipfzejpifjiprfhurehurghurhugerhuiogehougzerhuogerhuogrhougrhuogrhuoigrehugregerrgegrgegr
KUBY_DOCKER_USERNAME: kuby
KUBY_DOCKER_PASSWORD: _XXXXXXXXXXX
KUBY_DOCKER_EMAIL: [email protected]
DIGITALOCEAN_ACCESS_TOKEN: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DIGITALOCEAN_CLUSTER_ID: XXXX-XXX-4beb-b27c-XXXXXX
DATABASE_PASSWORD: XXXXXXX
from kuby-core.
camertron
commented on August 27, 2024
Thanks @rept, that's helpful. How many nodes do you have in your cluster?
from kuby-core.
rept
commented on August 27, 2024
@camertron Only one.
from kuby-core.
camertron
commented on August 27, 2024
@rept ok, I was finally able to reproduce the issue! It looks like the problem lies with Kubernetes v1.20, which has removed the selfLink field. I believe kubedb, the Kubernetes operator kuby uses to stand up a database, is expecting this field to exist. I'm sure new versions of kubedb have stopped using that field, but unfortunately AppsCode now requires you to obtain a (free) license, meaning I'm going to have to figure out how to work that in to the installation process somehow before kuby can make use of the new versions. In the meantime, I would suggest downgrading to Kubernetes 1.19. Unfortunately with DigitalOcean that means destroying and re-creating your cluster. After downgrading to v1.19, I was able to successfully deploy as normal.
from kuby-core.
rept
commented on August 27, 2024
@camertron thx for the help. After deleting the cluster it now works with v1.19.
from kuby-core.
Related Issues (20)
- Write docs for authoring providers
- Support for other lang apps HOT 3
- Rollback does not work: undefined method `metadata` HOT 1
- bundle install fails HOT 2
- [Feature Request] Add support for hetzner cloud HOT 1
- Can't deploy HOT 3
- Adding the gem hangs bundler forever HOT 3
- Mutiple hostnames? HOT 2
- Add support for multiple hostnames HOT 6
- Unable to build HOT 4
- [Feature Request] Add support for Citus HOT 3
- [Feature Request] Add support for Openshift HOT 1
- [Feature Request] Please add CloudNativePG as a PostgreSQL db
- Feature Request: support for Google Cloud - GKE as a provider HOT 2
- Kuby for a dockerized project HOT 1
- Cannot load bin/kuby in v0.20.0 HOT 2
- Assets & migration HOT 3
- Allow disabling of automatic migrations
- getkuby.io is down HOT 5
- kuby-core (0.20.1) raise error on dotenv (3.0.1) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kuby-core.