**Project Enhancement and Infrastructure Optimization**

[codebanesr]

DevOps Tasks:

1. API Gateway and Service Mesh:

   • Task: Implement Kong API Gateway for managing authentication.
   • Task: Integrate Linkerd service mesh for service communication, signed certificates, etc.

2. Logging and Monitoring:

   • Task: Enable Loki for log management.
   • Task: Configure Prometheus for monitoring.

3. Autoscaling:

   • Task: Implement Horizontal Pod Autoscaler (HPA) for backend services.
   • Task: Integrate KEDA or HPA for Selenium and Celery.

4. Helm Chart:

   • Task: Convert all components (Kong, Linkerd, Loki, Prometheus, etc.) to Helm charts for better manageability.

Application Tasks:

1. Celery Job Queues:
   • Task: Divide Celery jobs into separate queues.
   • Subtask: Assign 'reindex' jobs to a low-priority queue.
   • Subtask: Assign 'knowledgebase extraction' jobs to a medium-priority queue.
   • Subtask: Assign 'qdrant ingestion' jobs to a designated queue.
   • Subtask: Assign 'web crawl' jobs to a medium-priority queue.

App Improvements:

1. Streaming and Limits:

   • Task: Enable streaming on the '/backend/send' API.
   • Task: Increase the word limit for chatbot messages.

2. Timeout Adjustment:

   • Task: Increase the timeout for chatbot messages to improve processing.

   @yagarwal-allo can you help me prioritise these tasks please ? Feel free to create tasks you would want to work with me on.