Amazon Web Scraping


#4

Amazon Web Services has an API (application program interface) for data querying.

You must request access to the service.


#5

Screen scraping is not permissable…sounds like what u r doing… we are not allowed to use software to determine the lowest price we are only allowed to determine the lowest “bucket”…


#6

Hey guys, really, thanks for all the answers.

Some say OK, some say no. Does anyone have a link to this policy?

Is the an Amazon.com policy or a seller central policy?


#7

You are allowed to do that via the API. Some of the APIs have what they refer to as “throttling” so you cannot request too much data too fast and bog down the system, but if the ability to do what you are doing is available by some crafty API programming, then why would it be prohibited??


#8

I think what you are talking about is not really referred to as “data scraping” since it runs via an API. “Data scraping,” in the sense that would be against Amazon policy, would be an app that visits the page of each product and extracts the name, description, bullet points, reviews, etc. by reading the HTML code of each page. I can see why something like that would be prohibited, not your app. All that info is readily available to developers and you can dn download sample files that do that.

-Rick


#9

I have a wholesale company that sells to several on line sellers, who apparently have programs that do the same thing your doing, as they always know exactly which items to purchase and which ones to pass on.

A programs like that would be very beneficial to my business, when distributors offer me tons of inventory, I could run it and narrow my choices to the better stuff.

Anyone know where I can get the program or what it’s called?


#10

I know I came across the actual limits expressed as mbits/s and number of records/minute. I do not remember exactly where it was, but it was in the AWS help files.

They also expressed a concern of having repeating processes (like inventory sych) only run a certain number of times per hour, or have a specific “rest” period between requests.

The data is out there. An AWS specialist should be able to help you.

As for web scraping… as long as you are not republishing the information, there is NO DIFFERENCE between a scraper and an automated browser. As long as you do not abuse the system (which can be construed as a DOS attack or wasteful of bandwidth which slows down other customers’ browsing experiences) then there isn’t much Amazon can do.

Obviously, if you have a Seller Account then that can always be terminated at a moment’s notice for any reason, so keep that in consideration. I’m sure somewhere there is a policy that forbids it. However, it is like reselling brand name restricted products; legally you can, Amazon just doesn’t like it and can prevent you from doing stuff on its website.


#11

On the bottom of the main page is this link [Conditions of Use|http://www.amazon.com/gp/help/customer/display.html/ref=footer_cou?ie=UTF8&nodeId=508088]. In the conditions of use it states: “You may use Amazon Software solely for purposes of enabling you to use and enjoy the Amazon Services as provided by Amazon, and as permitted by the Conditions of Use, these Software Terms and any Service Terms.”

My little mind thinks this means that data scraping is technically not allowed. I do believe, however, that it is often done (on Amazon and elsewhere), but is rarely discussed because: a) it is kinda against the rules, and b) it give you an edge over the competition and you don’t want them to know about it.

Practically speaking if you do it in a manner that does not cause stress on the servers, no one will know or care. If you do otherwise, you may get a cease and desist letter in the mail.

Edited by: Better For Life on Apr 18, 2014 8:15 PM

Edited by: Better For Life on Apr 18, 2014 8:15 PM


#12

So long as your scraper software uses the api (you will know because it requires amazon api username and password), then it’s OK.

The software should not make more the one request per second, so you should find that a list of 100 asins takes approx 100 seconds to process.

This kind of app is commonly used by amazon associates, not so common to hear of it being used by sellers.


#13

To answer your question The Music Connection Wholesale, about where to get this software.

The original question poster did mention that they had this program built.

I used to use this kind of software a lot, but as an amazon associate not as a seller. I found that there were no really good reliable apps to do what I wanted so I hired a guy from rent-a-coder to build a custom app for me. My requirement was to return title, description, image, upc, weight, price etc all from a list of supplied asins.

It was easy enough to find a programmer experienced in Amazon AWS (as it was at the time, I think they call it something else now) and build an app. The trouble is that it had to be updated a couple of times as the AWS spec can change a bit every year.

Basically, you probably won’t find a commercial app. You will need to hire a coder.


#14

Thank you.


#15

>Recently, I had a program built for me.
>
> I upload UPC’s, it gives me back
>
> Asin
> Title
> Lowest New Price
> Buy Box Price
> Sales Rank
>
> I understand this is data scraping, and recently read that it is against Amazon policy. Can someone point me to where this information can be found.
>
> I recently created a case and asked if data scraping is restricted, they replied “no restrictions for data scraping.”
>
> Any help on the matter would be appreciated. Thank you in advance.
"
Many online merchants, including Amazon, Walmart and Target, expressly prohibit companies from accessing their pricing data by “robot” or other data mining tools. Some may go so far as to expressly prohibit competitors from using data they see for competitive purposes. Amazon’s site says:

“Amazon or its content providers grant you a limited, non-exclusive, non-transferable, non-sublicensable license to access and make personal and non-commercial use of the Amazon Services. This license does not include any resale or commercial use of any Amazon Service, or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of any Amazon Service or its contents; any downloading or copying of account information for the benefit of another merchant; or any use of data mining, robots, or similar data gathering and extraction tools.”
"

[Window Shopping Felonies - FierceRetailIT|http://www.fierceretail.com/retailit/story/window-shopping-felonies#ixzz2zIb2m9g7]


#16

y


#17

Yet they do it themselves, or they pay a “contractor” to do it for them. How else do they monitor other’s prices?

A roomful of 10,000 maniacs at computers?


#18

I know right?!?!

Just goes to show that what’s good for the goose isn’t allowed for the gander…

I use VBScript to automate IE to do an XML request to my webserver that triggers a php script to send me an email that makes google voice dial my phone to call me as an alarm clock and the message returned from my server is read by a JavaScript loop that waits for a readystate change from the request then appends the document body in the DOM with an activeX object to communicate through my parallel port to a 120v relay that starts my coffee and flashes “GOOD MORNING” with pictures of kittens on my monitor every day :wink:


#19

But - and I am only saying this because I care - there are a lot of decaffeinated brands on the market today that are just as tasty as the real thing. :wink:


#20

Indeed,
And flammable corn syrup solids that got stuck in the strainer at the soda factory fortified with titanium white fillers instead of cream too :wink:

As long as my “swiched2Coffee” doesn’t say “back2Beer” I’m good with almost any drink…

I went through much aluminum prior to realizing I was merely recycling my childhood with every crushed can…

Things said about others is one thing but, looking in the mirror & realizing I’d become one of the people I would’ve talked about really woke me up…

I’ve been in a life long love-hate relationship with my surviving neurons…


#21

Collecting data using Amazon’s own AWS Api is acceptable, using a robot or web scraping software is not acceptable.

AWS is smart enough to only allow you limited access , hence the 1 request per second rule. Web scrapers use resources and are very different to Api scraping.


#22

Is the web API free or does it come with upgraded pkgs like webstore, Amazon payments, etc.

Most things like that are packaged and sold by the company that holds the data…

Web scraping is always free and it gets data from a company without paying for it
in a way that is not unlike putting OTA TV captured with antennas and using it as a content source -

There’s no problem with that tech if the TV corps get their palms greased a second time for what they already decided was a reasonable enough profit to “broadcast to the public”…


closed #23