Feature board for ERPNext

Migrating board.castlecraft.co.in
Updating DocTypes for frappe        : [========================================] 100%
Updating DocTypes for erpnext_feature_board: [========================================] 100%
Updating Dashboard for frappe
Updating Dashboard for erpnext_feature_board
Building search index for board.castlecraft.co.in
Retrieving Routes                   : [==                                      ] 5%
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/commands/migrate.py", line 51, in <module>
    main()
  File "/home/frappe/frappe-bench/commands/migrate.py", line 44, in main
    migrate_sites()
  File "/home/frappe/frappe-bench/commands/migrate.py", line 35, in migrate_sites
    migrate()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 95, in migrate
    build_index_for_all_routes()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/search/website_search.py", line 128, in build_index_for_all_routes
    return ws.build()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/search/full_text_search.py", line 37, in build
    self.build_index()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/search/full_text_search.py", line 96, in build_index
    writer = ix.writer()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/whoosh/index.py", line 464, in writer
    return SegmentWriter(self, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/whoosh/writing.py", line 515, in __init__
    raise LockError
whoosh.index.LockError

Seems to be during migration from containers.

• Web view (also allowed for Guest users)

  • show paginated Improvement list
  • on Improvement Form show PR url, status, and site name.
  • add a form/popup to initiate a "Moderation Request" referencing Improvement
  • keep track of "Moderation Request" by user on the form view

• Webhooks

  • Required for automation after github PR is updated and developer fixes code after user review
  • Webhooks also adds "Moderation Request", to upgrade
  • Open PR and PR with active sites only, add "Moderation Request" on change of PR (commits, force-push)

• "Moderation Request" Types

  • Initiate site for improvement
  • Upgrade site for improvement
  • Delete site for improvement
  • Add user (and set roles) in site for Improvement (use admin password)

• Moderators can:

  • Respond to "Moderation Request", "act" on it or "deny/close"
  • Add user and set roles for the sites generated for PR
  • Trigger builds for PR
  • Trigger updated after PR is updated
  • Trigger tear down, if site generation results in Error, or PR is merged.
  • check error logs and report

• Add moderator on invite

  • share invitation link, expires in configurable time
  • Filter by Domain
  • Access Improvements by domain

Rename "Moderation Request" to something more meaningful.

Check if it is error with setup or error with PR

https://discuss.erpnext.com/t/how-to-improve-user-involvement-in-development-of-erpnext/75708/76?u=revant_one

Error Log

Traceback (most recent call last):
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/base_document.py", line 363, in db_insert
    ), list(d.values()))
  File "/home/revant/frappe-bench/apps/frappe/frappe/database/database.py", line 146, in sql
    self._cursor.execute(query, values)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/revant/frappe-bench/env/lib/python3.7/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1062, "Duplicate entry 'feat-tax-deduction-against-advance-payments' for key 'PRIMARY'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/revant/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 100, in execute_job
    method(**kwargs)
  File "/home/revant/frappe-bench/apps/erpnext_feature_board/erpnext_feature_board/erpnext_feature_board/doctype/github_repository/github_repository.py", line 66, in create_repository_improvements
    update_improvement(pull, repository)
  File "/home/revant/frappe-bench/apps/erpnext_feature_board/erpnext_feature_board/erpnext_feature_board/doctype/github_repository/github_repository.py", line 87, in update_improvement
    improvement_doc.save()
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 284, in save
    return self._save(*args, **kwargs)
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 306, in _save
    self.insert()
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 252, in insert
    raise e
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/document.py", line 249, in insert
    self.db_insert()
  File "/home/revant/frappe-bench/apps/frappe/frappe/model/base_document.py", line 376, in db_insert
    raise frappe.DuplicateEntryError(self.doctype, self.name, e)
frappe.exceptions.DuplicateEntryError: ('Improvement', 'feat-tax-deduction-against-advance-payments', IntegrityError(1062, "Duplicate entry 'feat-tax-deduction-against-advance-payments' for key 'PRIMARY'"))

• This app will run in Kubernetes cluster
• Cluster will host
  • container registry
  • mariadb
  • nfs-server
  • redis
  • Helm Operator
• kaniko will build and push the docker images
• Install the builds with helm chart (helm operator), use custom values
• wildcard subdomain ingress to a domain. (cert-manager+letsencrypt+dns01)

--context=git://github.com/frappe/frappe_docker.git
--context-sub-path=images/worker
--build-arg=FRAPPE_REPO=https://github.com/frappe/frappe
--build-arg=ERPNEXT_REPO=https://github.com/frappe/erpnext
--build-arg=BENCH_REPO=https://github.com/frappe/bench
--build-arg=FRAPPE_VERSION=v13.25.0
--build-arg=ERPNEXT_VERSION=v13.25.1
--build-arg=PYTHON_VERSION=3.9
--target=erpnext

• [x] Only allow Add test user for "Ready" improvements self.validate()
• [x] Schedule Cron to delete Approved "Build", "Delete", "Upgrade"
• [x] On "Build", "Delete", "Upgrade" Review Request for selected Improvement, Once the Improvement state is changed, change the state of all other same requests of same type on same improvement.
• [x] Delete and Upgrade only allowed for Ready
• [x] Build only allowed for No Deployment Status On improvement

import requests
import json

s = requests.Session()
s.post(f"{i.site_url}/api/method/login", data=json.dumps({"usr":"Administator","pwd":f"{i.site_admin_password}"}))
user = {"email":"[email protected]", "first_name":"User", "new_password":"hello@123", "send_welcome_email":0, "roles":[{"role":"System Manager"}]}
s.post(f"{i.site_url}/api/resource/User", data=json.dumps(user)).json()

• No ui available to trigger the builds.
• for initial testing bench console was used. and there are whitelisted functions in improvement.py that need to be called from frontend.

• site_url: data
• site_admin_password: data (needs to be visible)
• deployment_status: select
  • Ready
  • Build Queued
  • Build Queue Failed
  • Building
  • Build Complete
  • Build Error
  • Release Queued
  • Release Queued Failed
  • Release Error
  • Upgrade Queued
  • Upgrade Queued Failed
  • Rebuilding
  • Rebuild Error
  • Rebuild Complete
  • Upgrading
  • Upgrading Failed
  • Upgrade Error
  • Delete Queued
  • Delete Queue Failed

States of improvement:

Ready: Set when Improvement is added

Build Queued

• Set by using whitelisted function.
• Picked up by process_build_queue_improvements

Build Queue Failed

• process_build_queue_improvements sets "Build Queue" Improvements to "Build Queue Failed" if calling the create_build_image_job throws exception
• Save Error Log

Building

• process_build_queue_improvements sets "Build Queue" Improvements to "Building" if create_build_image_job response is successful.

Build Complete

• process_building_improvements sets "Building" Improvements to "Build Complete" if get_job_status() returns job is complete.

Build Error

• process_building_improvements sets "Building" Improvements to "Build Error" if get_job_status() returns exception.

Release Queued

• process_build_complete_improvements sets "Build Complete" Improvements to "Release Queued" if create_helm_release() returns success.
• set site credentials on Improvement

Release Queued Failed

• process_build_complete_improvements sets "Build Complete" Improvements to "Release Queued Failed" if create_helm_release() returns exception.

Release Error

• process_release_queued_improvements sets "Release Queued" to "Release Error" if get_helm_release_status() returns exception

Ready

• process_release_queued_improvements sets "Release Queued" to "Ready" if get_helm_release_status() returns exception

Upgrade Queued

• Set by user by whitelisted function call, allow only if Improvement has "site credential data"

Upgrade Queued Failed

• process_upgrade_queued_improvements sets "Upgrade Queued" to "Upgrade Queue Failed" if create_build_image_job() returns exception

Rebuilding

• process_upgrade_queued_improvements sets "Upgrade Queued" to "Rebuilding" if create_build_image_job() returns success

Rebuild Error

• process_rebuilding_improvements sets "Rebuilding" to "Rebuild Error" if get_job_status() returns failure

Rebuild Complete

• process_rebuilding_improvements sets "Rebuilding" to "Rebuild Complete" if get_job_status() returns success

Upgrading

• process_rebuild_complete_improvements sets "Rebuild Complete" to "Upgrading" if update_helm_release() returns success

Upgrading Failed

• process_rebuild_complete_improvements sets "Rebuild Complete" to "Upgrading Failed" if update_helm_release() returns error

Upgrade Error

• process_upgrading_improvements sets "Upgrading" to "Upgrade Error" if get_helm_release_status() returns failure

Ready

• process_upgrading_improvements sets "Upgrading" to "Ready" if get_helm_release_status() returns success

Delete Queued

• Set by user through whitelisted function

Delete Queue Failed

• process_delete_queued_improvements sets "Delete Queued" to "Delete Queue Failed" if tear_down_improvement_site_resources returns error

[doc.remove]

• process_delete_queued_improvements deletes "Delete Queued" if tear_down_improvement_site_resources returns error

On error options:

• contact admin
• reset button, cleanup and teardown.

List of cron functions to add

• process_build_queue_improvements
• process_building_improvements
• process_build_complete_improvements
• process_release_queued_improvements
• process_upgrade_queued_improvements
• process_rebuilding_improvements
• process_rebuild_complete_improvements
• process_upgrading_improvements
• process_delete_queued_improvements