We need an experienced Python developer with expertise in web scraping, machine learning, and automation to review, optimize, and run an existing domain crawler. The goal is to improve efficiency, accuracy, and reliability.
This project will be conducted in two stages:
Stage 1: Successfully Run the Existing Crawler Your first task is to set up and successfully run the crawler to validate that you can: - Install and configure all dependencies - Execute the crawler on a dataset of domain names - Generate valid output - Troubleshoot any initial setup issues
Once this is successfully demonstrated, we will move to Stage 2.
Stage 2: Propose a Better Solution or Manage Ongoing Runs - Depending on your expertise and analysis of the current crawler, you will either:
A) Propose & Implement Improvements (Rewrite or Optimize the Crawler) - Identify inefficiencies and suggest a better architecture - Improve performance & accuracy (e.g., async scraping, multiprocessing, better ML model) - Ensure scalability if running on large datasets - Refactor code for long-term maintainability OR B) Run & Maintain the Crawler Monthly - Execute the crawler on a scheduled basis - Monitor accuracy and adjust as needed - Fix bugs & ensure stability - Implement new features & improvements over time - Provide regular reports on data quality and insights
About the current crawler The current crawler is built in Python and classifies domain names based on their content (e.g., parked, active, expired). It utilizes: - Requests / Aiohttp (Web scraping) - Selenium (Browser automation) - BeautifulSoup (HTML parsing) - Langdetect / NLP (Language detection & text classification) - XGBoost (Machine learning classification)
Ideal Candidate: - Strong Python skills (async programming, OOP) - Web scraping & automation expertise (requests, Selenium, BeautifulSoup) - Machine learning experience (XGBoost, Scikit-learn, NLP) - Familiarity with databases (SQL/NoSQL) for storing results - Cloud/DevOps experience (if needed for large-scale deployment) - Ability to work independently and ensure high-quality results - High level in English
Project Timeline & Budget: Deliverables expected within 2-4 weeks Budget: Open to discussion based on experience & estimated work required.
Boost Google Maps Local Ranking Category: Analytics, Competitor Analysis, Content Strategy, Internet Marketing, Keyword Research, Link Building, Search Engine Marketing (SEM), SEO Budget: $30 - $250 CAD
07-Nov-2025 11:03 GMT
Framer Site Content Migration Category: CSS, Framer, HTML, JavaScript, UI / User Interface, Web Design, Web Development Budget: $10 - $70 USD
07-Nov-2025 11:02 GMT
National Education ERP Leads Category: Business Development, Customer Service, ERP Software, Lead Generation, Market Research, Marketing, Sales, Sales Management, Sales Promotion Budget: ₹12500 - ₹37500 INR
07-Nov-2025 11:02 GMT
Fix Google Index & Sitemap Category: Content Strategy, Data Analysis, Search Engine Marketing (SEM), SEO, Web Development, Web Security, Website Management, Website Optimization Budget: ₹600 - ₹1500 INR
Impactful Posters, Logo & Digital Design Category: Digital Design, Graphic Design, Illustration, Logo Design, Photoshop, Poster Design, UI / User Interface, Visual Design Budget: $250 - $750 USD
07-Nov-2025 10:57 GMT
Modern Apartment Renovation Renders Category: 3D Animation, 3D Design, 3D Modelling, 3D Rendering, 3D Visualization, 3ds Max, Architectural Rendering, Blender, Home Design, Interior Design Budget: €30 - €250 EUR
07-Nov-2025 10:55 GMT
Script Translator & Multitask Support ayna Category: C, Programming, Coding, Data Entry, Data Processing, English (US) Translator, English Translation, Python, Script Writing, Translation, Typing Budget: ₹750 - ₹1250 INR
07-Nov-2025 10:55 GMT
Digital Twin Research for 5G/6G Category: Docker, MATLAB, Network Administration, Python, Simulation, Technical Writing, Telecommunications Engineering, Wireless Budget: ₹12500 - ₹37500 INR
07-Nov-2025 10:54 GMT
Recruit Territory Business Managers Nationwide Category: Business Analysis, Business Writing, Human Resources, Interviewing, Report Writing, Research, Sales, Talent Acquisition Budget: ₹12500 - ₹37500 INR
