Import new WoS data from ZXDB?

This thread is intended to discuss my long time offer to import ZXDB data content into Lee's new WoS site, and address any objections posted against this idea. This way, we won't hijack any other thread into this kind of discussion.

Notice this is not a new idea. As a matter of fact, helping rebuild WoS was exactly the reason I started working on ZXDB. As I wrote in my first post about ZXDB at July 11, 2016 (immediately after Lee decided to quit): "By providing a new "clean" database, I'm hoping to reduce the burden on whoever assumes responsibility over WoS development afterwards."

If the new WoS site imports data from ZXDB (instead of importing from old WoS archive), it will benefit from almost 1 year of hard work on fixing, updating and improving the original WoS archive, with contributions from several community members. ZXDB is an open iniciative, every single change applied to the original archive is kept available in a package file called "ZXDB_increments.zip" that's always available for download. It's a good evidence that a huge amount of time has been invested on it already. Moreover, ZXDB is already being used by different sites (especially http://spectrumcomputing.co.uk/ and http://incubator.kolbeck.dk/ ) so we know it works very well in practice. Finally, every change made on the database was carefully validated to avoid introducing errors or loosing any content from the original WoS archive, thus absolutely everything was preserved. As a matter of fact, all this effort on double-checking and cross-validating every ZXDB change has even allowed me to find a few errors in ZXSR (already fixed now) and a few errors in the WoS files repository.

IMHO there's no good reason to ignore all this hard work and restart WoS from scratch, with all the archive inconsistencies that have been already fixed!


PS: Of course I don't mean any criticism against Martijn or anybody else. Sites like WoS and ZXSR are an excellent work! However ZXDB benefits from the structure of a modern database, so I could use automatic consistency checks and easily cross-reference data in ways Martijn had to do manually, so it got much easier for me to identify problems and fix them.
Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
Thanked by 3p13z jonesypeter hikoki
«134

Comments


  • [This is a reply to this post, that I will answer here to avoid hijacking another thread:]

    I already asked you several times what data you have changed and why, and you've refused to answer.

    Every time you asked, I gave you basically the same answer, reproduced below.

    There were literally over a thousand fixes in ZXDB. All these fixes fall into 3 categories:

    1. Data inconsistent or absurd. For instance, search Infoseek for "Alvaro Mangado Cruz" and "Alvaro Mandago Cruz", you will find both (inconsistency). Notice that http://www.worldofspectrum.org/infoseekid.cgi?id=0017812 is supposed to run in a Spectrum 126 model (absurd).

    2. Data referenced by WoS but missing in WoS. For instance http://www.worldofspectrum.org/infoseekid.cgi?id=0009227 has link to "Hack Attack 3" that doesn't exist in WoS.

    3. Inaccurate or incomplete data. For instance WoS says number of players in Bruce Lee is "2-simultaneous" but it's both "simultaneous" and "alternating", depending on the option you choose.

    Besides database fixes, the following improvements have been also made in ZXDB:

    1. In ZXDB, several "lost" files (stored but unindexed inside old WoS FTP archive) were also organized and properly associated with their corresponding database entries.

    2. ZXDB also merged data from SPOT/SPEX. For instance, the old WoS archive contained less than 3,700 magazine references to games, and was able to locate another 120,000 magazine references from SPOT/SPEX related to games. All this information was fully integrated and organized into ZXDB, so now there are 168,500 magazine references associated to database entries (mostly games) in ZXDB.

    3. ZXDB was also integrated with the ZXSR site. Whenever a magazine review about a game exists in ZXSR, the magazine reference in ZXDB will also contain a link to the corresponding ZXSR page. This information was incomplete in old WoS archive, now it's 100% updated.

    4. ZXDB also reorganized information about each database entry, whenever it made sense. For instance, next ZXDB update will have a distinction between game title and bibliographic title, which means titles like "The Party is Open - Part 1" will display correctly instead of http://www.worldofspectrum.org/infoseekid.cgi?id=0027087 . Comments with references to other games now contain their InfoseekID to eliminate ambiguities in case of 2 games with same title and publisher. More information about each game was also added, such as links to the corresponding game page in Mobygames when available, development roles for each developer (loading screen, in-game graphics, code, music, etc). And so on...

    5. ZXDB also contains entries for games already stored in the old WoS FTP site but "lost" without an InfoseekID (i.e. not catalogued). It also contains most of the latest games released after WoS stopped updates. There are over 2,000 entries (mostly games) that didn't exist in the old WoS archive.

    Now you are saying you've added things to the database without discussion.

    No, everything I added to ZXDB follows exactly the same criteria adopted by Martijn and you. Let's see:

    1.) Information about ZX81 programs already existed in WoS. For instance this page contains 2 titles for the ZX81. In ZXDB, I just added the titles that were missing (although their files were already stored at WoS FTP site).

    2.) Information about programs that only work in other "Speccy-like" (non Sinclair) machines already existed in WoS, for instance Power Up (Pentagon 128) and SQ Tracker (Scorpion). In ZXDB, I followed exactly the same rule, when I added more machines exactly in the same situation.

    3.) Information about unfinished titles already existed in WoS, as you can see in this list. In ZXDB, I followed exactly the same rule, when I added information about Double Dragon Redux. Moreover, graphics from Ice Hockey and Iron Sword already existed in WoS FTP site but they were not properly indexed. In ZXDB, I just organized them by fixing this problem.

    This is why I kept asking what you had done that was different to the data I am using.

    And this is why I kept answering whenever you asked.

    If you want to know about every individual change that has been made, that's fine. All this information is available in a package file called "ZXDB_increments.zip", that contains every SQL statement that was executed to improve the database. This file contains over 530,000 lines of SQL queries (aprox. 120Mb of SQL text) including comments. It was a huge amount of work, and anyone is welcome to validate and "audit" all the work that has been done.

    It certainly means a lot of work has been done, but every one of those changes made to the original archive had a very good reason. I didn't just start changing everything randomly. If you are frustrated that you want to know everything that has been changed, but it would take you too long to read everything, it's not my fault. It's like "ZXDB_increments.zip" is the "source code" of the work done in ZXDB, and "ZXDB_latest_mysql.sql" is the final "compiled" version. If you were a Linux user, it would be like demanding to understand everything in the Linux operating system before accepting to use it, claiming the source code is not good enough, and keep asking Linux developers for a detailed description about every detail that makes it different from Windows...
    Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
  • Most of the items you've listed we picked up on import anyway. The majority of fixes are to magazines that weren't indexed initially, and based on the work Gerard has already done for WoS.

    Since you keep pushing the point, and I have no time to go through thousands of lines of "fixes" that only you have verified and could just as easily be wrong, and you aren't happy to wait for me to get time, then I have to say no. We won't be using ZXDB. In the time it takes to check everything it will have been finished anyway.
    My test signature
    Thanked by 1hikoki
  • While I don't want to get into argument in any direction, and I see work being done on both sets of data, I'd like to point out that there is a good side to having two separate sets of fixes:

    After the new infoseek is up, it should be relatively(tm) easy to compare the two sets of data to find mismatches to find things that may need to be resolved manually (to one, or both, of data sets). This should result in a better quality data set than either set has on its own.
    http://iki.fi/sol | http://iki.fi/sol/speccy/ | https://github.com/jarikomppa/speccy
    http://goo.gl/q2j0NZ - make ZX Spectrum choose your own adventure games, no programming needed (release 3)
  • Sol_HSA wrote: »
    While I don't want to get into argument in any direction, and I see work being done on both sets of data, I'd like to point out that there is a good side to having two separate sets of fixes:

    After the new infoseek is up, it should be relatively(tm) easy to compare the two sets of data to find mismatches to find things that may need to be resolved manually (to one, or both, of data sets). This should result in a better quality data set than either set has on its own.

    That's a good point :)
    My test signature
  • Just tried ZXDB at http://spectrumcomputing.co.uk/ and I like it.
    The archive contains the latest titles, search response is very fast, so far it looks really good.
    Great job, Einar. :)
    Thanked by 1jonesypeter
  • Pegaz wrote: »
    Just tried ZXDB at http://spectrumcomputing.co.uk/ and I like it.
    The archive contains the latest titles, search response is very fast, so far it looks really good.
    Great job, Einar. :)

    Do not forget the fact that the search actually works! As it stands, this is a very important bragging point in today's Speccy world! OK, that was perhaps a bit too easy I admit.

    All that's left is one more natural step hmmm 8->
    Thanked by 1jonesypeter
  • Not being dependent on a single site / database is indeed a plus.
    Each site will always offer a different browsing experience, and that's good.
    I use WOS for many things, like links to C64 versions or listing games using 3D vector graphics, etc.

    If we look at the big picture, there are even other gaming sites with Spectrum content, like www.mobygames.com.
    I have no clue if their content is original or not, and if it could be used to cross check any entries.
  • The SpectrumComputing site, though excellent in its information gathering, is lacking in presentation imho. It's heavily mobile-centric, several scroll-pages worth for each entry, information spaced out too broadly. Anyways, sorry, the presentation does not appeal to me :-( Aside from that, it does a fine job.
    What now?
  • My test signature
  • F_Clowder wrote: »
    The SpectrumComputing site, though excellent in its information gathering, is lacking in presentation imho. It's heavily mobile-centric, several scroll-pages worth for each entry, information spaced out too broadly. Anyways, sorry, the presentation does not appeal to me :-( Aside from that, it does a fine job.

    As an aside, websites now have to be mobile centric or they get punished by Google. Mobile usage is a lot more than PC usage.
    My test signature
  • Just detect what type of device the user is on. It shouldn't have to be mobile-centric by default, right?
    What now?
  • That's just a lot of unused space!!! :((
    What now?
  • edited May 2017
    So, I checked both sites out on my fisher-price mobile, and right now the ZXDB one is more appealing on mobile. But my previous experience with it on the PC still stands.

    PS To both Einar and Fogartylee ... I'm so glad I'm not the one doing the design work! It's evidently not an envious task at all, to please both mobile and desktop viewers.
    Post edited by F_Clowder on
    What now?
  • F_Clowder wrote: »
    So, I checked both sites out on my fisher-price mobile, and right now the ZXDB one is more appealing on mobile. But my previous experience with it on the PC still stands.

    PS To both Einar and Fogartylee ... I'm so glad I'm not the one doing the design work! It's evidently not an envious task at all, to please both mobile and desktop viewers.

    I haven't done the mobile versions of the tables yet. Besides, with the new API, there's nothing to stop you doing your own design with the WoS data :)
    My test signature
  • I'm not touching that job description with a fourteen foot barge pole! :))

    I'd rather enjoy the fruitage of your hard labor
    What now?
  • F_Clowder wrote: »
    Just detect what type of device the user is on. It shouldn't have to be mobile-centric by default, right?

    That's a lot of devices to detect these days. It's much more common now to design for mobile and build up (although clearly I haven't!)

    My test signature
  • It's only really just two categories of devices, mobile vs desktop. Shouldn't be rocket science compared to feat you've already accomplished, right?

    Then again, I have no clue, so I have the benefit of complete ignorance and can easily simplify the process in my mind's eye!
    What now?
  • edited May 2017
    One of my suggestions for Spectrumcomputing was just hide divs without content. I think this way will make info easy to wade through for the user.
    As for mobile versus desktop, see the code of this page: https://speccy21.updog.co/qaop/qaop_Covertape#2128k.html
    I think I managed to place a css style which adapts to any size
    Post edited by hikoki on
  • Maybe it's because it's late, but aside from a couple of tweaks, I'm not seeing any problem with either layout.
    My test signature
  • I haven't done the mobile versions of the tables yet. Besides, with the new API, there's nothing to stop you doing your own design with the WoS data :)

    What new API? Did I miss something? :S
  • Maybe it's because it's late, but aside from a couple of tweaks, I'm not seeing any problem with either layout.

    Is the presentation of the data part of "the database"? Most technical discussions would keep the two things completely separate and Einar specifically started the thread for discussion of the data. So aren't we wandering off topic?

    Then again, Lee gave what appeared to be the definitive answer yesterday: no.
  • Is the presentation of the data part of "the database"? Most technical discussions would keep the two things completely separate...
    Hell yes!

    If presentation is not part of the database design, then you will always end up with either "design by committee", or "overengineered" or "select over 500 options before you finally find the game you want".

    Then again, most people design databases never care about use cases. Then again "Most technical discussions" seems to be great in theory. :)
  • Well... if you really want to get into it, some one needs to write a front end that gets data from both ZXDB and new WoS, it can then display the data side by side in a easy to read format ;-)

    Mark
  • Timmy wrote: »
    Is the presentation of the data part of "the database"? Most technical discussions would keep the two things completely separate...
    Hell yes!

    Hmmm, OK. That's not the way I've ever seen it done. I've always seen the database designed to hold the necessary data and be able to query it so as to yield the result data sets users are going to want. (Assuming a transactional installation, obviously.)

    I've never come across the idea that a database is designed such that queries to it might be rendered on, say, a mobile phone as opposed to a computer screen. That would be a job for the presentation layer.

    But OK, I was just afraid the rather important discussion was wandering off topic. :)
  • 1024MAK wrote: »
    Well... if you really want to get into it, some one needs to write a front end that gets data from both ZXDB and new WoS, it can then display the data side by side in a easy to read format ;-)

    Mark
    +1 I suggested Einar the same yesterday, a visual diff web frontend for Lee to audit easily his work. I'm afraid he would need a hand though.
  • edited May 2017
    F_Clowder wrote: »
    It's only really just two categories of devices, mobile vs desktop. Shouldn't be rocket science compared to feat you've already accomplished, right?

    Then again, I have no clue, so I have the benefit of complete ignorance and can easily simplify the process in my mind's eye!

    If only it were as simple as that :-) Mobile you've got titchy 4" screens on phones up to 10"+ on tablets - and all of them have varying resolutions from 800x400 up to 1080p and some even with 4K. Desktops you've got some nutters still using 1024x768 all the way up to 4K and the website is expected to cope with them all perfectly.

    It's a bloody nightmare... Bootstrap helps a great deal but does, as you've noticed, introduce a hell of a lot of white space on larger screens unless you're really careful. Mobile-first is the mindset you have to employ when creating websites these days. It's both great and a pain in the arse.

    And I'm with thricenightly - I've never seen presentation be a factor in database design. That's the job of the presentation layer, not the database.
    Post edited by Vampyre on
  • edited May 2017
    Take a look at my sample above.
    Cannot one define the amount of white space between elements according to a range of display resolutions?
    Post edited by hikoki on
  • Some PC nutters (me!) still use 1024x600 on their dinky laptops....
    I am from that generation, when Pluto was still a planet, and the Earth is round.

    Bomb Munchies on WOS thread
    Bomb Munchies Ver1930 17th Nov 2017 (look for the blue download box ) If you get a time-out message and live in the UK then try after 9pm-3am.
    Send me a PM and I can email it to you too. Kent, UK
  • MatGubbins wrote: »
    Some PC nutters (me!) still use 1024x600 on their dinky laptops....

    You are, unfortunately in a minority. Maybe its time for an upgrade
  • Doesn't matter if he does or not. With bootstrap he'll just be seeing the ipad display :)
    My test signature
Sign In or Register to comment.