revant / erpnext_feature_board Goto Github PK

• No ui available to trigger the builds.
• for initial testing bench console was used. and there are whitelisted functions in improvement.py that need to be called from frontend.

Error Log

Traceback (most recent call last):
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/base_document.py", line 363, in db_insert
    ), list(d.values()))
  File "/home/revant/frappe-bench/apps/frappe/frappe/database/database.py", line 146, in sql
    self._cursor.execute(query, values)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1062, "Duplicate entry 'feat-tax-deduction-against-advance-payments' for key 'PRIMARY'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/revant/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 100, in execute_job
    method(**kwargs)
  File "/home/revant/frappe-bench/apps/erpnext_feature_board/erpnext_feature_board/erpnext_feature_board/doctype/github_repository/github_repository.py", line 66, in create_repository_improvements
    update_improvement(pull, repository)
  File "/home/revant/frappe-bench/apps/erpnext_feature_board/erpnext_feature_board/erpnext_feature_board/doctype/github_repository/github_repository.py", line 87, in update_improvement
    improvement_doc.save()
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 284, in save
    return self._save(*args, **kwargs)
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 306, in _save
    self.insert()
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 252, in insert
    raise e
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 249, in insert
    self.db_insert()
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/base_document.py", line 376, in db_insert
    raise frappe.DuplicateEntryError(self.doctype, self.name, e)
frappe.exceptions.DuplicateEntryError: ('Improvement', 'feat-tax-deduction-against-advance-payments', IntegrityError(1062, "Duplicate entry 'feat-tax-deduction-against-advance-payments' for key 'PRIMARY'"))

Migrating board.castlecraft.co.in
Updating DocTypes for frappe        : [========================================] 100%
Updating DocTypes for erpnext_feature_board: [========================================] 100%
Updating Dashboard for frappe
Updating Dashboard for erpnext_feature_board
Building search index for board.castlecraft.co.in
Retrieving Routes                   : [==                                      ] 5%
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/commands/migrate.py", line 51, in <module>
    main()
  File "/home/frappe/frappe-bench/commands/migrate.py", line 44, in main
    migrate_sites()
  File "/home/frappe/frappe-bench/commands/migrate.py", line 35, in migrate_sites
    migrate()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 95, in migrate
    build_index_for_all_routes()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/search/website_search.py", line 128, in build_index_for_all_routes
    return ws.build()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/search/full_text_search.py", line 37, in build
    self.build_index()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/search/full_text_search.py", line 96, in build_index
    writer = ix.writer()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/whoosh/index.py", line 464, in writer
    return SegmentWriter(self, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/whoosh/writing.py", line 515, in __init__
    raise LockError
whoosh.index.LockError

Seems to be during migration from containers.

• site_url: data
• site_admin_password: data (needs to be visible)
• deployment_status: select
  • Ready
  • Build Queued
  • Build Queue Failed
  • Building
  • Build Complete
  • Build Error
  • Release Queued
  • Release Queued Failed
  • Release Error
  • Upgrade Queued
  • Upgrade Queued Failed
  • Rebuilding
  • Rebuild Error
  • Rebuild Complete
  • Upgrading
  • Upgrading Failed
  • Upgrade Error
  • Delete Queued
  • Delete Queue Failed

keeping it in event hook will cause the .save() to fail even in case of discourse site network errors

The Github PAT is not necessary for public repositories

• This app will run in Kubernetes cluster
• Cluster will host
  • container registry
  • mariadb
  • nfs-server
  • redis
  • Helm Operator
• kaniko will build and push the docker images
• Install the builds with helm chart (helm operator), use custom values
• wildcard subdomain ingress to a domain. (cert-manager+letsencrypt+dns01)

• build frappe image(s)
• build erpnext image(s) based on frappe image(s)
• use the ERPNext images to install site with frappe PR.

Thoughts?

States of improvement:

Ready: Set when Improvement is added

Build Queued

• Set by using whitelisted function.
• Picked up by process_build_queue_improvements

Build Queue Failed

• process_build_queue_improvements sets "Build Queue" Improvements to "Build Queue Failed" if calling the create_build_image_job throws exception
• Save Error Log

Building

• process_build_queue_improvements sets "Build Queue" Improvements to "Building" if create_build_image_job response is successful.

Build Complete

• process_building_improvements sets "Building" Improvements to "Build Complete" if get_job_status() returns job is complete.

Build Error

• process_building_improvements sets "Building" Improvements to "Build Error" if get_job_status() returns exception.

Release Queued

• process_build_complete_improvements sets "Build Complete" Improvements to "Release Queued" if create_helm_release() returns success.
• set site credentials on Improvement

Release Queued Failed

• process_build_complete_improvements sets "Build Complete" Improvements to "Release Queued Failed" if create_helm_release() returns exception.

Release Error

• process_release_queued_improvements sets "Release Queued" to "Release Error" if get_helm_release_status() returns exception

Ready

• process_release_queued_improvements sets "Release Queued" to "Ready" if get_helm_release_status() returns exception

Upgrade Queued

• Set by user by whitelisted function call, allow only if Improvement has "site credential data"

Upgrade Queued Failed

• process_upgrade_queued_improvements sets "Upgrade Queued" to "Upgrade Queue Failed" if create_build_image_job() returns exception

Rebuilding

• process_upgrade_queued_improvements sets "Upgrade Queued" to "Rebuilding" if create_build_image_job() returns success

Rebuild Error

• process_rebuilding_improvements sets "Rebuilding" to "Rebuild Error" if get_job_status() returns failure

Rebuild Complete

• process_rebuilding_improvements sets "Rebuilding" to "Rebuild Complete" if get_job_status() returns success

Upgrading

• process_rebuild_complete_improvements sets "Rebuild Complete" to "Upgrading" if update_helm_release() returns success

Upgrading Failed

• process_rebuild_complete_improvements sets "Rebuild Complete" to "Upgrading Failed" if update_helm_release() returns error

Upgrade Error

• process_upgrading_improvements sets "Upgrading" to "Upgrade Error" if get_helm_release_status() returns failure

Ready

• process_upgrading_improvements sets "Upgrading" to "Ready" if get_helm_release_status() returns success

Delete Queued

• Set by user through whitelisted function

Delete Queue Failed

• process_delete_queued_improvements sets "Delete Queued" to "Delete Queue Failed" if tear_down_improvement_site_resources returns error

[doc.remove]

• process_delete_queued_improvements deletes "Delete Queued" if tear_down_improvement_site_resources returns error

On error options:

• contact admin
• reset button, cleanup and teardown.

List of cron functions to add

• process_build_queue_improvements
• process_building_improvements
• process_build_complete_improvements
• process_release_queued_improvements
• process_upgrade_queued_improvements
• process_rebuilding_improvements
• process_rebuild_complete_improvements
• process_upgrading_improvements
• process_delete_queued_improvements

• Only allow Add test user for "Ready" improvements self.validate()
• Schedule Cron to delete Approved "Build", "Delete", "Upgrade"
• On "Build", "Delete", "Upgrade" Review Request for selected Improvement, Once the Improvement state is changed, change the state of all other same requests of same type on same improvement.
• Delete and Upgrade only allowed for Ready
• Build only allowed for No Deployment Status On improvement

Referring this PR https://github.com/frappe/erpnext/pull/24603 it is going into version-13-hotfix instead of develop branch.

This information is needed to base the build on the branch given here.

for now using it from json.loads(doc.raw_data).get("base", {}).get("ref")

Cron to delete merged improvements

• Queue site Delete first
• Once site is deleted, Delete the improvement and related Review requests

There's a function that already handles this in the code. Just need to wire it up with cron in hooks.

Check if it is error with setup or error with PR

https://discuss.erpnext.com/t/how-to-improve-user-involvement-in-development-of-erpnext/75708/76?u=revant_one

• Web view (also allowed for Guest users)

  • show paginated Improvement list
  • on Improvement Form show PR url, status, and site name.
  • add a form/popup to initiate a "Moderation Request" referencing Improvement
  • keep track of "Moderation Request" by user on the form view

• Webhooks

  • Required for automation after github PR is updated and developer fixes code after user review
  • Webhooks also adds "Moderation Request", to upgrade
  • Open PR and PR with active sites only, add "Moderation Request" on change of PR (commits, force-push)

• "Moderation Request" Types

  • Initiate site for improvement
  • Upgrade site for improvement
  • Delete site for improvement
  • Add user (and set roles) in site for Improvement (use admin password)

• Moderators can:

  • Respond to "Moderation Request", "act" on it or "deny/close"
  • Add user and set roles for the sites generated for PR
  • Trigger builds for PR
  • Trigger updated after PR is updated
  • Trigger tear down, if site generation results in Error, or PR is merged.
  • check error logs and report

• Add moderator on invite

  • share invitation link, expires in configurable time
  • Filter by Domain
  • Access Improvements by domain

Rename "Moderation Request" to something more meaningful.

• Web portal view for users.
• Block password on web view.
• Or skip web views and only stick to desk app.

open for discussion.