My site is on a webhosting vps account over at webhostpython com
My website is a book deals website, which lists the best book deals every day
I find approx 1/3rd of the deals listed on a 3rd party book site: website 'x'
I need a scraper that will visit website 'x', grab its list of books (and some details), then visit each book's Amazon page, and scrape each book's details from Amazon
The scraper will then drop a csv file on my server with ALL the scraped details (from both sites)
In Summary, there are 3 steps:
# 1 visit website 'x', collect book details
#2 visit Amazon pages for each book and scrape Amazon book details
#3 create updateable csv file on server
THEN
Ensure WP All Import captures details/fields correctly (see below, most of this is already done)
IN DETAIL:
:
#1 Website 'x' does not have an RSS feed but updates daily.
= therefore, when the scraper/script visits the site it must make a record of ALL books scraped so it will ONLY scrape new books (today's books) each day.
NOTE: Website 'x' is a privately owned website, I do not want to hammer their resources or be flagged in any way, EG: Be respectful, I don't have permission to scrape it.
#2 The site is only a 'listing' of books, with links to book pages on Amazon, so the scraper must go to the site, collect links to Amazon, then visit those links and scrape each Amazon page.
+ scraper will need to visit multiple pages on the site
I have manually captured the required details from BOTH site 'x' and Amazon and created a sample csv file which you can use = all columns correctly labelled etc
I will provide EXACT details of what to scrape, with marked up screenshots, all communications will be very clear and easy to understand.
I don't mind what platform is used to build the scraper/submitter (php, python, etc) so long as it scrapes and submits.
WP All Import:
If you have experience with WP All Import, that is good, but probably not necessary. I have been able to figure out 90% of the necessary settings.
There are a couple of settings I cannot figure out, which maybe need a php function added, I would like you to check/edit the settings to ensure the import runs smoothly/correctly.
EG #1 I can't figure out how to match categories from csv to categories on my site!
EG #2: A custom field needs to display a date, and I don't know how to do that
EG#3: Some book titles need small text changes such as this title:
'Hunt for Justice Box Set: Books 1–2'
= IF a book title contains a colon ':' THEN remove the colon and ALL that follows it
So, title will now display as: 'Hunt for Justice Box Set'
ALSO
= IF a book title contains the words 'Box Set' THEN change 'Box' to 'Boxed' and surround words with brackets
So, title will now display as 'Hunt for Justice (Boxed Set)
ALSO
= If a title contains the words 'Omnibus', or 'Complete Omnibus', or 'The Complete Ominibus' THEN change to 'Omnibus' and surround with brackets
So, title will display as 'book title (Omnibus)
= everything else I have done.
Additional info:
I can provide some additional precise details to the person who creates this. I have worked with a few developers for this kind of scraper / submitter, so I already know most of the details / issues which relate to it so this is a very simple job for someone who knows what they are doing.
Future Projects:
There are also another 3-5 sites I want to scrape in the same way, so there may be additional similar projects for the right worker.
Please note, these are book sites, they are not high-earners. I do not have large budgets for any of my work. So I am looking for low cost solutions. But I write very good reviews for good workers!
Please write to me with an accurate assessment of how much time this would take you, when you can start and how much you will charge. I will ensure I provide as many details as possible, so the job doesn't have any 'unexpected surprises'.
Thank you.
Budget: $50
Posted On: February 19, 2021 10:17 UTC Category: Data Extraction Skills:Web Scraper, Python, PHP, Data Scraping
Skills: Web Scraper, Python, PHP, Data Scraping Country: United Kingdom
