About the Role

We're looking for a skilled Software Developer with a strong focus on web crawling to join our team. You’ll be responsible for building and maintaining scalable, reliable crawlers to gather data from social media platforms, news websites, and other public sources. Your work will be central to our data-driven applications, so attention to detail, adaptability, and clean code are key.

What You’ll Do

• Design, develop, and maintain robust web crawlers using Python.
• Crawl and extract structured/unstructured data from social media, news websites, and dynamic content sources.
• Handle pagination, rate limits, CAPTCHAs, and other anti-bot mechanisms.
• Store and manage large datasets efficiently using relational or NoSQL databases.
• Monitor crawler performance and log issues, and ensure data integrity.

Requirements

• 2+ years of experience in software development with a strong command of Python.
• Hands-on experience with web crawling libraries such as Scrapy, BeautifulSoup, Selenium, or Playwright.
• Solid understanding of HTTP, web scraping best practices, and handling various content types (JSON, HTML, XML, etc.).
• Experience working with databases (e.g., PostgreSQL, MongoDB, Redis).
• Experience working with message queues, especially RabbitMQ, for task distribution and scaling crawlers.
• Familiarity with crawling social media platforms (e.g., Twitter/X, Reddit, Instagram, TikTok) and news/media sites.
• Experience working with proxies, rotating user agents, and headless browsers
• Strong debugging and problem-solving skills.

Why Join Us?

• Competitive salary and benefits package.
• Comprehensive health insurance coverage to ensure your well-being.
• Flexible working hours and remote work options.
• Daily lunch cart allowance to enjoy a variety of meals.
• Opportunity to shape the future of our data infrastructure.