Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 3 of 3
  1. #1
    Regular Coder
    Join Date
    Jan 2008
    Posts
    130
    Thanks
    0
    Thanked 0 Times in 0 Posts

    how does sites that do price comparison's get the info ?

    Hello all
    how does sites that do price comparison get there info about the prices and such?
    does they use the sites web services api ? or they use web spiders?

  • #2
    Master Coder
    Join Date
    Jun 2003
    Location
    Cottage Grove, Minnesota
    Posts
    9,502
    Thanks
    8
    Thanked 1,089 Times in 1,080 Posts
    Some do "scraping" ... which isn't ethical...
    Others probably subscribe to some database(s) which provide the data?

    If a site offers an RSS feed, that would be an ethical way to get the data.

    I would guess that the type of website (what they sell) would be
    a key thing on what is available.

    You can do a search on Google for "web page scraping" to see how you
    can make your own spider. Like I said before, it's not really ethical to steal
    bandwidth from another site without permission.

  • #3
    Mega-ultimate member
    Join Date
    Jun 2002
    Location
    Winona, MN - The land of 10,000 lakes
    Posts
    1,855
    Thanks
    1
    Thanked 45 Times in 42 Posts
    You just sign up for multiple datafeeds and integrate them into your system. For example, if you sign up for...

    Amazon.com
    Buy.com
    Shopzilla

    You're going to have data for any product from at least 3, and probably dozens of sites.

    The tricky part is setting up the database to handle that much data.

    I've already integrated amazon into one of my sites (http://www.yourlistonline.com) and plan on pulling in buy.com and shopzilla in the near future.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •