We need an experienced Python developer with expertise in web scraping, machine learning, and automation to review, optimize, and run an existing domain crawler. The goal is to improve efficiency, accuracy, and reliability.
This project will be conducted in two stages:
Stage 1: Successfully Run the Existing Crawler Your first task is to set up and successfully run the crawler to validate that you can: - Install and configure all dependencies - Execute the crawler on a dataset of domain names - Generate valid output - Troubleshoot any initial setup issues
Once this is successfully demonstrated, we will move to Stage 2.
Stage 2: Propose a Better Solution or Manage Ongoing Runs - Depending on your expertise and analysis of the current crawler, you will either:
A) Propose & Implement Improvements (Rewrite or Optimize the Crawler) - Identify inefficiencies and suggest a better architecture - Improve performance & accuracy (e.g., async scraping, multiprocessing, better ML model) - Ensure scalability if running on large datasets - Refactor code for long-term maintainability OR B) Run & Maintain the Crawler Monthly - Execute the crawler on a scheduled basis - Monitor accuracy and adjust as needed - Fix bugs & ensure stability - Implement new features & improvements over time - Provide regular reports on data quality and insights
About the current crawler The current crawler is built in Python and classifies domain names based on their content (e.g., parked, active, expired). It utilizes: - Requests / Aiohttp (Web scraping) - Selenium (Browser automation) - BeautifulSoup (HTML parsing) - Langdetect / NLP (Language detection & text classification) - XGBoost (Machine learning classification)
Ideal Candidate: - Strong Python skills (async programming, OOP) - Web scraping & automation expertise (requests, Selenium, BeautifulSoup) - Machine learning experience (XGBoost, Scikit-learn, NLP) - Familiarity with databases (SQL/NoSQL) for storing results - Cloud/DevOps experience (if needed for large-scale deployment) - Ability to work independently and ensure high-quality results - High level in English
Project Timeline & Budget: Deliverables expected within 2-4 weeks Budget: Open to discussion based on experience & estimated work required.
Expand Global Tech Talent Network Category: Admin Support, Artificial Intelligence, Business Analysis, Database Development, Game Development, Machine Learning (ML), Mobile App Development, Solutions Architecture, Testing / QA, Web Development Budget: min ₹5000000 INR
05-Oct-2025 22:01 GMT
Army Values Briefing Presentation Category: Content Development, Content Writing, Education & Tutoring, Graphic Design, Instructional Design, Powerpoint, Presentations, Public Speaking, Report Writing, Research Writing Budget: $30 - $250 USD
05-Oct-2025 22:00 GMT
Automated Excel CFSI Dashboard Category: Data Analysis, Data Entry, Data Management, Data Processing, Data Visualization, Excel, Excel Macros, Excel VBA, Microsoft Access, Visual Basic Budget: $10 - $300 USD
05-Oct-2025 22:00 GMT
Tableau Sales Growth Dashboard Category: Analytics, Business Intelligence, Data Analysis, Data Integration, Data Management, Data Processing, Data Visualization, Report Writing, Tableau Budget: $15 - $25 USD
Boost Google Maps Visibility Regional Category: Content Marketing, Content Writing, Google Ads, Google Analytics, Google Maps API, Internet Marketing, Keyword Research, Link Building, Search Engine Marketing (SEM), SEO Budget: $250 - $750 USD
05-Oct-2025 21:54 GMT
Chinese Novel & Web Fiction Translator (Chinese to English) Category: Editing, English (US) Translator, English Translation, Language Tutoring, Proofreading, Traditional Chinese (Hong Kong), Traditional Chinese (Taiwan), Translation Budget: $15 - $25 USD
05-Oct-2025 21:54 GMT
Recover Overwritten ZIP Ubuntu Category: Digital Forensics, Linux, Network Administration, System Admin, Ubuntu Budget: ₹600 - ₹1500 INR
05-Oct-2025 21:53 GMT
Results-Focused Google Ads Audit + Managment Category: Conversion Rate Optimization, Data Analysis, Digital Marketing, Google Ads, Google Adwords, Google Analytics, Internet Marketing, Keyword Research, Search Engine Marketing (SEM), SEO Budget: min $50 AUD
05-Oct-2025 21:53 GMT
Hairbrush STL with Heart Shape Cap that opens Category: 3D Animation, 3D CAD, 3D Design, 3D Drafting, 3D Layout, 3D Modelling, 3D Printing, 3D Rendering, AutoCAD, Solidworks Budget: $10 - $30 USD
05-Oct-2025 21:52 GMT
AutoCAD Basketball Game Design Category: 3D Design, 3D Modelling, 3D Rendering, AutoCAD, Building Architecture, CAD / CAM, Solidworks, Sports Design Budget: $30 - $250 USD
