Job Title: LLM - DevOps Developer
Experience: 5+ Years
Type: Remote
Timings: 8 hours/day (Mandatory: 6 PM – 11 PM IST, Remaining 3 hours flexible)
Join Date: Immediate Joiner
Job Overview:
We are seeking an experienced DevOps Developer with exposure to Large Language Models (LLMs) such as ChatGPT or Gemini. This role supports one of the foundational LLM companies in enhancing model performance through evaluation, prompt engineering, scripting, and DevOps infrastructure management.
Key Responsibilities:
Design and develop prompts related to DevOps and Infrastructure topics.
Write executable scripts to validate model responses to prompts.
Conduct model evaluations (Evals) and analyze benchmark data.
Evaluate AI model outputs, rank them, and provide detailed feedback.
Contribute to the generation of SFT and RLHF data.
Collaborate with cross-functional teams including researchers and annotators.
Participate in peer reviews of code and documentation.
Continuously improve tools, strategies, and evaluation processes.
Mandatory Skills:
DevOps Tools: 4+ years of experience with tools like Terraform and Ansible.
Containers: 1+ year with Docker and Kubernetes.
Version Control: 1+ year with GitHub/GitLab.
Scripting: Proficient in Python or Bash.
LLM Exposure: Experience working with tools like ChatGPT, Gemini, or other LLM-based applications (no need to have built LLMs).
Nice-to-Have:
Experience in SFT and RLHF data generation.
Familiarity with CI/CD tools like Jenkins, CircleCI.
Strong understanding of AWS or other cloud environments.
Knowledge of system monitoring, security, and automation best practices.
Exposure to evaluation strategies for AI systems.
Requirements:
5+ years of overall experience in DevOps or infrastructure roles.
Excellent problem-solving and collaboration skills.
Strong English communication (written and verbal).