WOS API WTF?!
I try to use the API.
It would seem that this is an honest way to get information, coined the owner database (site).
But in fact it is almost useless.
Enough to make about 10 requests and speed of issue falls heavily.
And as soon as you banyat 20-30 queries.
The question is - why do we need such an API?
I do not believe that is the protection against DDOS. If someone wants to DDOS WOS he does not need to be API.
I think that the owner just greedy. But what he is entitled to be greedy information that we have collected all!
It turns out that we are limited to our own information.
What I did:
15 requests from 1.1 second intervals. Then pause 7 seconds.
30-50 requests I have been banned.
It would seem that this is an honest way to get information, coined the owner database (site).
But in fact it is almost useless.
Enough to make about 10 requests and speed of issue falls heavily.
And as soon as you banyat 20-30 queries.
The question is - why do we need such an API?
I do not believe that is the protection against DDOS. If someone wants to DDOS WOS he does not need to be API.
I think that the owner just greedy. But what he is entitled to be greedy information that we have collected all!
It turns out that we are limited to our own information.
What I did:
15 requests from 1.1 second intervals. Then pause 7 seconds.
30-50 requests I have been banned.
Post edited by newart on
Comments
What exactly are you trying to do? I hope it's not to download games wholesale because that *will* be a bad thing to do because the bandwidth is limited.
It's an information tool. It's currently being used in emulators and apps to download games and other related content in a sane way.
Pot. Kettle. Black. WoS is 100% donation and payment free. Martijn runs the website and all other supporting paraphernalia out of his own pockets. He has the right to decide what's good for the website and its users.
Bytes:Chuntey - Spectrum tech blog.
Otherwise, Martin would have published specific limitation.
Or posted to the database txt.
Once this is done ... suggests only one conclusion.
Nowhere is it stated that I can not use it for their own purposes.
When it comes to cleanliness, the WOS very lazily updated. A lot of games have the status of Missing in Action - in this case they can be found in Google...
Code:
[PHP]OpenConsole()
wait = 1
nm = 0
skip = 600
If ReadFile(0, "spectrum4ever.org.htm")
While Eof(0) = 0
t$ = ReadString(0)
nm + 1
If nm > skip
title$ = StringField(t$, 2, "/")
id$ = StringField(t$, 1, "/")
xml$ = SimpleHTTP_GET( URLEncoder( "http://www.worldofspectrum.org/api/infoseek_search_xml.cgi?title="+title$) )
Print(Str(nm) +" - "+ title$)
id_wos$ = cut(xml$, "<id>", "</id>")
If xml$ And id_wos$
Delay(3000+Random(1000))
xml$ = SimpleHTTP_GET( URLEncoder( "http://www.worldofspectrum.org/api/infoseek_select_xml.cgi?id="+id_wos$) )
If xml$
CreateFile(1, "XML/"+id$+".xml")
WriteString(1, xml$)
Print(" - "+id_wos$)
Else
PrintN(" - BAD INFO :(")
EndIf
PrintN("")
Delay(3000+Random(1000))
Else
PrintN(" - BAD ID :(")
EndIf
wait + 1
If wait = 29+Random(10)
Delay(15000+Random(5000))
wait = 0
EndIf
EndIf
Wend
CloseFile(0)
EndIf[/PHP]
Everything I do is a request to the API.
If you already have all the games then why are you trying to access 20-30 at a time?
I suspect you're doing something you're not telling us.
I suspect that you have a paranoia. ;)
There is a project spectrum4ever.org - there collected cassette versions of games and programs. Cracked by various hacker (Bill Gilbert, etc.)
Initially, the database information was collected only the name of the game and the name of hackers.
Users are asked to develop the site and expand the information about the games, adding the name of the game author, date of manufacture, etc.
The games themselves with WOS I do not need.
All I do is take a list of name of the game as it is in spectrum4ever.org - are extracted from the resulting XML ID games and downloading page with its description.
That's it. Nothing tricky or illegal.
I'm in no hurry. But I have to pick up an experimental pause.
Restart my router that would change my IP when I WOS bans
and so on.
This is stupid.
Something like this?
Why not help out the archive team by posting those links or sending it to them vial mail? If they are genuine I'm sure they will be interested in filling the gaps in the archive.
Bytes:Chuntey - Spectrum tech blog.
Do not see the point.
WOS will still infinitely far from that fullness which could have.
For example, we all know that at the Spectrum in Russia was released nearly 2,000 electronic journals. All of them are in the public domain. You just need to take and dowload.
But the author WOS it does not. It's laziness? Indifference? I do not know, but this attitude does not motivate me to help.
He ran this site alone for 15 years putting thousands of hours into it. He did it all for free, never earning a penny from it. He paid for servers for all these years from his own wallet.
Okay, he currently neglected updating WOS which worries me much :( I guess he has a lot of duties but still... Martijn, please change the job if it doesn't leave you time for anything else, we need you and you'll be a happier man ;)
As for Russian e-journals it's not just downloading them, it's again a lot of works with adding them to database, creating entries, authors, descriptions etc. It may be done one day but there is always an queue of another things to do and besides any Russian knows where to get them already and almost any non Russian is unable to read them, so it isn't high priority
As for API, I'm not competent to discuss, so I'll leave it to other
The problem many times with the russian ZX material is the language. Many games, or other Spectrum stuff is hosted in russian sites or written directly in russian, making it hard for non-russian speakers, to gather and process it.
For example, there is a lot of Spanish members who are taking care of the Spanish ZX material, as many times the language (Spanish) is a problem as well.
If more russian were involved, I guess there would be more russian stuff here on WoS :)
when was the last time you bought something and submitted it here ? e.g. a cover or MIA.. if you don't even do that, then that is even worse.. and I should complain about your lack of action ;)
Martjin does a good job.. heck one day I might even buy him an amstel (I still find it funny a dutchman is bigging up spectrum when it's a c64 heartland ;) )
Well, then don't help. As if you had ever thought of helping anyway.
Laziness? Indifference? I think you're talking about yourself. So lazy that you complain about an API to download a database instead of making your own. Indifference about all those Russuan stuff that even if you know them you don't tell us. Nice attitude you have there.
Games List 2016 - Games List 2015 - Games List 2014
do you mean got ban?
MIA's can't be found on Google, or they wouldn't be MIA!
Well, some (or many) MIA's have been recently preserved and offered for download, but the status hasn't changed on WoS. They're just waiting for the next TZX Vault or SPA2 update.
- IONIAN-GAMES.com -
When I open many links as new tab in my browser I always get banned to 24h sooner or later. And I really need this for information scavenging - so there is obviously something not right with WoS anti-ddos shield.
Did you get in contact with Martijn about this? If there is an issue on his end, then surely he can assist you with it.
I downloaded once whole archive through ftp and hadn't any problems.
Strange thing with this banning
For those who can't add extra contribution (I've a few fanzines to add but no scanner!) - it would be nice to put something towards the costs.
Of course not.
I'm talking about things in the status of MIA!
If the base was in the public domain. I would download one file.
In place of this I have to do 3600 requests.
Do you seriously think that Martin saves the traffic?
I have my base. A lot of bases.
Open your eyes: zxpress.ru, zxtunes.com, spectrum4ever.org, zxaaa 3 until the non-publication of the project.
I'll tell you a secret, to work for the Spectrum, I spend more time than most of the work.
I do not mean to brag, after all I'm doing this because I like it and I believe that this should be done.
Nevertheless, it gives me a right to criticize, Martin as the administrator, I'm not talking about him as a person.
That's not true.
You versed in websites? Do you know something about databases?
Add one database to another, even if there are 100,000 records - it is a matter of one or two evenings. It's not hand labor.
That is not true ;)
This is the business rules, the rules of etiquette, etc.
Sorry for misunderstanding, newart.
Based on an earlier discussion on adding a feature to infoseek on who worked for which company I got the impression that WoS uses a 'hand-coded' database. Some of the 'inconsistencies' that have been reported on infoseek strengthen this assumption since they simply cannot happen in a normal database.
In which case it smells a lot like manual labor. ;-)
Shifting everything into a 'normal' standard database should, for someone with the required skillset, be a walk in the park.
That is not the question I asked.