New WoS and ZXDB

2»

Comments

  • Right, that's enough, now I want to see Einar's post!!

    And no, I'm not being sarcastic, I really wanna see cos I'm a shallow twat that waits on forum posts hehehe
    So far, so meh :)
  • And I'm a tw*t!!! :D
    So far, so meh :)
  • edited July 4
    Multiple sources have confirmed the same words from Lee Fogarty posted at his own private Facebook group, that I cannot access. I wasn't planning to quote him, but since this is now in the open, there's no reason for privacy concerns anymore. He apparently forgot to post his claims in his own forum, so let me help him:
    Lee Fogarty: "I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
    From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
    Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
    That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
    WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
    This is all something that could have been sorted with a PM."

    So that's the main point. Is new WoS simply using the same data from old WoS that was imported into ZXDB? Or is it using such an early version of ZXDB (from July/August 2016) that only contained old WoS data, so there's no need to credit ZXDB (despite literally about 50,000 fixes I did when importing this data)?

    Unfortunately the answer is no. To understand the difference, let's take a look at ZXDB chronology. The summary below has plenty of links to prove everything, although I suggest ignoring the links for now and just reading from start to finish:

    So that's the point. It comes back to something I wrote in my original post:
    I imported old WoS content with their help from July 2016 to August 2016. If they had used one of the early versions of ZXDB that they participated, without crediting ZXDB, I would leave it alone. However they chose to use a version of ZXDB from September 2018, in order to take advantage of over 2 years of other people's work, without crediting anybody. That's a ZXDB version released 2 years after WoS stopped supporting ZXDB and started attacking my work. About 1 year after ZXDB and SpectrumComputing were censored at WoS thus forcing ZXDB to move to another forum. Months after I was personally censored at WoS without ever receiving any explanation.

    Everybody is probably asking now, how do we know that new WoS is using ZXDB 1.0.8? Is it really much different from the original data from old WoS?

    I'm glad you asked :)

    I will have to get technical now, but I will explain it so everyone can understand. We will compare new WoS content against old WoS and a few ZXDB versions. Let's see what happens!

    To reproduce this experiment at home, you need MySQL (or even better MariaDB) and any SQL client (HeidiSQL, MySQL Workbench, DBeaver, etc). They are open source and free. Also download a few versions of ZXDB from Github (click on "commits" to find and download older versions) and load one of them into your database.

    I already mentioned you could visit new WoS software page and click on "EXPORT CSV (ALL)", to download some of the data from all titles stored in new WoS. I know lots of people did it (you may still have an old copy of this file yourself, perhaps in your Recycle Bin?). Let's start with "software-20200616.csv" from a day after new WoS was launched (we will talk about files from different days later).

    Here's a database script to import this file into a database. Even if you don't know SQL, you should be able to see it's quite straightforward:
    create table x_newwos (
      rows0 varchar(100),
      id int(11) not null primary key,
      title varchar(500),
      slug varchar(500),
      no_players varchar(100),
      turn_type varchar(100),
      entry_type varchar(100),
      availability varchar(100),
      comments varchar(5000),
      is_x_rated varchar(100),
      is_crap varchar(100),
      clone_of varchar(100),
      old_id int(11),
      title_publisher varchar(500),
      publishers varchar(500),
      all_publishers varchar(500),
      entry_groups varchar(500),
      distribution_status_type varchar(500),
      display_image varchar(500)
    );
    
    load data local infile 'software-20200616.csv'
      into table x_newwos character set utf8
      fields terminated by ',' optionally enclosed by '"'
      lines terminated by '\n' ignore 1 lines;
    

    Now download the original Martijn's WoS internal file "maindb.dat". Hopefully Lee Fogarty declared it "open source" so I don't need to worry anymore about sharing it. If you don't believe this file is authentic, choose any game at random and compare the corresponding line in this file against the old WoS pages. Let us know if you spot any difference!

    Here's a simple database script to import this file:
    create table x_entries (
      titlekey varchar(500),
      pubkey varchar(500),
      title varchar(500),
      release_year varchar(10),
      orig_publisher varchar(500),
      re_publishers varchar(500),
      memory varchar(500),
      players varchar(500),
      joysticks varchar(500),
      genre varchar(500),
      category varchar(500),
      language varchar(500),
      distrib_status varchar(500),
      schemetype varchar(500),
      downloads varchar(500),
      flags varchar(500),
      authors varchar(500),
      aliases varchar(500),
      id int(11) primary key not null,
      spot_num varchar(500),
      spot_genre varchar(500),
      spot_full_price varchar(500),
      spot_budget_price varchar(500),
      spot_disk_price varchar(500),
      spot_comments varchar(500),
      spot_publisher varchar(500),
      license varchar(500),
      groupname varchar(500),
      comments varchar(5000),
      series varchar(500),
      orig_price varchar(500),
      c64_ref varchar(500),
      spanish_price varchar(500),
      wikipedia varchar(500),
      typein_ref varchar(500),
      authoring varchar(500)
    );
    
    load data local infile 'maindb.dat'
      into table x_entries character set utf8
      fields terminated by '\t'
      lines terminated by '\n';
    

    The first CSV file from new WoS didn't have much useful content besides title, original publisher, and comments. Comparing title and original publisher from old titles won't help, since old WoS rarely got this information wrong so it almost never changed. However comparing comments is very useful, since they are continuously improved in ZXDB with fixes, further details, etc.

    Here's a simple SQL to compare comments (except backslashes) between 2 tables. Notice it only compares titles that existed in old WoS (i.e 24369 titles with ID below 28187) to give new WoS a better chance:
    select e.id,e.comments,x.comments from entries e
    inner join x_newwos x on e.id = x.old_id
    where replace(coalesce(e.comments,''),'\\ ',' ') <> replace(coalesce(x.comments,''),'\\ ',' ')
    and e.id <= 28187;
    

    From this comparison, you will get the following results:
    new WoS (software-20200616.csv) vs. old WoS (maindb.dat)        - 2583 differences
    new WoS (software-20200616.csv) vs. ZXDB 1.0.0 (April 2018)     - 5 differences
    new WoS (software-20200616.csv) vs. ZXDB 1.0.8 (September 2018) - 0 (zero) differences
    new WoS (software-20200616.csv) vs. ZXDB 1.0.9 (October 2018)   - 1 difference
    new WoS (software-20200616.csv) vs. ZXDB 1.0.69 (latest)        - 766 differences
    

    As you can see, there's a lot more similarity between new WoS and current ZXDB, than between new WoS and old WoS.

    What if you want to repeat this test yourself to believe it, but you only have a newer CSV file from a different day? No problem. Although the CSV format at new WoS has changed over time, any CSV file downloaded before 2 days ago (when all comments changed into a bloody mess) will do. You just need to add or remove a couple columns from the import script, based on the column names you can see at the top of your CSV file. For instance, here's the same script adapted according to the CSV columns from 2 days ago:
    create table x_newwos (
      rows0 varchar(100),
      id int(11) not null primary key,
      title varchar(500),
      slug varchar(500),
      no_players varchar(100),
      turn_type varchar(100),
      entry_type varchar(100),
      availability varchar(100),
      comments varchar(5000),
      is_x_rated varchar(100),
      is_crap varchar(100),
      clone_of varchar(100),
      old_id int(11),
      title_publisher varchar(500),
      release_year varchar(10),
      search_title varchar(500),
      known_errors text(30000),
      has_inlay varchar(4),
      has_loading_screen varchar(4),
      machine_type varchar(500),
      publishers varchar(500),
      control_types varchar(500),
      theme varchar(500),
      all_publishers varchar(500),
      machine_types varchar(500),
      entry_groups varchar(500),
      az varchar(500),
      distribution_status_type varchar(500),
      display_image varchar(500),
      index x_id(old_id)
    );
    
    load data local infile 'software-20200702.csv'
      into table x_newwos character set utf8 
      fields terminated by ',' optionally enclosed by '"'
      lines terminated by '\n' ignore 1 lines;
    

    The same comparison using a more recent CSV file will show nearly identical results, except for 2 titles: Reckless Rufus (new comments added on June 26th) and Werner's Quest (new comments added on June 17th but later lost).

    It's absolutely clear that new WoS is really using content taken from ZXDB, not from old WoS. Instead of just pointing a few examples, we have now executed a comparison involving all titles. Even better, I provided instructions so anyone can replicate this experiment at home to see by themselves. And this comparison demonstrated that new WoS content is very much different from old WoS, not so much different from current ZXDB, and absolutely identical to ZXDB from September 2018.

    As promised, this is my final post providing evidences. There's no need to prove anything else.

    So what now? Well, THAT QUESTION will require one more post. But it's late, so let's talk about that tomorrow.


    NOTE: On June 18th, I did a similar test at SpectrumComputing and found 2572 differences (instead of 2583) between new WoS and old WoS. It's because I compared new WoS against old Wos data that was already converted to ZXDB in 2016. This new comparison now, directly between new WoS and old WoS, is even more accurate and indicates even more differences.
    Post edited by Einar Saukas on
    Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
  • UPDATE: Please download "maindb.dat" from here.
    Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
  • Same old same old tiresome posts. We get it. Theres some crossover in the data. Who really cares? There's half a dozen of you squabbling in the background going "he said", "she said" while most of the rest of us just get on with it. It's really a shame that you've let it get to you like this.
  • edited July 5
    Same old same old tiresome posts. We get it. Theres some crossover in the data. Who really cares?

    More than you'd think care. There's a world of difference between using the data (all who update ZXDB having stated often they're happy should WoS do so) and accusing Einar of being a thief, stating several times there's no intention of using ZXDB's data, forking said data into your own database and then actively trying to hide it when called out about it. That's just reprehensible behaviour, with the subsequent ignoring of it being the crowning turd on this entire sh!t-fest.
    Post edited by Vampyre on
  • We get it. Theres some crossover in the data. Who really cares?
    If you think it is about "crossover in the data", then you don't "get it".
    I care.
    And this thread was naturally, reasonably and peacefully finished right there - until you commented.
  • I get it, I just don't think it matters to most people using the site. Many won't know, many won't care, it's just politics that has nothing to do with the speccy. Whether the site runner is a dodgy geezer or not, has no impact on my use of the forums and the archive. Of the hundreds/thousands of users of both sites, there's a tiny minority involved in this argument.
  • Many won't know
    Well that's the problem isn't it. Lee can continue to lie about it here and on the Facebook group and 90% of people will never see this thread, so go on believing that he's the saviour of WoS and that Einar is another of those dastardly trolls...

    Except pandora's box is open now. People are going to keep pointing it out so the "ignore it and hope it goes away" approach isn't going to work.
    How the times have changed. Now the Elite Viper ships are descending on Lee like there's no tomorrow :))
    My rubbish website including the redrawn Amstrad schematics and the new home of the Sinclair FAQ wiki.
  • guesser wrote: »
    Now the Elite Viper ships are descending on Lee like there's no tomorrow :))
    :)) :)) :))
    Oh, no. Every time you turn up something monumental and terrible happens.
    I don’t think I have the stomach for it.
    --Raziel (Legend of Kain: Soul Reaver 2)

    https://www.youtube.com/user/VincentTSFP
  • both sides DID build a HUGE database
    its OUR database.
    btw, old promise, if the spam on my hotmail account would be GONE i said to re-grant spectrum-computing?
    well, its amazing but althose "latino spam" that came from "no where" after my one big confrontation, it is GONE..... did some one keep a promise? then i can.

    and if you use a line or picture or music from some one MENTION FROM WHO
    becouse thats is HONEST even while the person says its 100% free, like me.
    oh, the database only could grow this big BY gathering EVERYTHING on one spot.
    my old website http://home.hccnet.nl/c.born/ has changed to http://www.cborn.nl/zxfiles/ so just click it and select a file
  • guesser wrote: »
    Many won't know
    Well that's the problem isn't it.
    Is it?

    I'm of the opinion it doesn't matter much.

    Lee isn't WoS, he's just a bloke who hosts it. Certainly no saviour, the archive has been static for years for no apparent reason.

    I'm personally more interested in what Einar produces and thinks of games than quibbles about who did what to a database. Einar may have a point, but that point has been repeatedly made, so much so that it comes across a bit obsessive now.
  • edited July 5
    SO WHAT NOW?

    I have a few requests to the WoS team. Feel free to point out if I'm asking anything unreasonable at all:


    • An apology for slandering me would be the honorable thing to do now. Frankly I'm not really expecting it to happen anymore, but I'm an optimistic person!

    • Please stop trying to disguise using ZXDB every time you are caught. It's not convincing anyone, it's just ruining everything. The line "Your honour, I'm refunding to banks the part of the money that the police has found, can I go now?" doesn't quite work. Now that it's proved new WoS is using ZXDB, anything else you change will be just "new WoS is using ZXDB but crippled". You won't get off the hook for using ZXDB dishonestly, and you will be providing a bad service for your own users. It's the worst of both worlds. A lot of users still visit WoS nowadays, either because they don't know any better, or because they just don't care. Everyone else that cares about your attitudes have already left, so you are just wasting your time trying to prove yourself to those left who obviously don't care. So please stop the whac-a-mole and focus your efforts to provide a decent service to your remaining users instead. Martijn spent decades building WoS credibility, his legacy deserves better.

    • Please stop trying to take credit for other people's work. Nobody is questioning who built the new WoS site and has been maintaining the WoS forum. Is it not enough to receive well deserved credit for those? However the database is not yours. The old WoS files were built by Martijn, maintained by him for decades with help from many contributors, then converted by myself into a proper database (with my questions answered mostly by Gerard and Martijn), then maintained by the ZXDB team for the last 4 years with help from many contributors. As I mentioned already, new WoS only uses the list of publishers you converted in 2017 (everything else is taken from ZXDB) and it still contains the same errors that I pointed out in 2016 and you claimed to have fixed in 2017. Removing even more improvements from ZXDB 1.0.8 won't solve anything and won't benefit anybody. Please do the right thing, credit ZXDB properly at new WoS, as a sign of respect for all the people that have been working hard to keep Martijn's initiative alive. Notice there's absolutely no need to credit me personally for anything, I never asked for it, just credit ZXDB to properly recognize everyone's effort, and let's get over it.

    • Please update new WoS to use latest version of ZXDB instead. Since you are already using it anyway, there's no point in keeping it outdated. It's bad for your own users to see wrong data already fixed everywhere else, and it's more work for the ZXDB team every time someone points out conflicting information between ZXDB and WoS, and we have to convince them that WoS is wrong.

    • Please respect the ZXDB license. It doesn't exist to take "ownership" over old WoS data or anything. It simply asks to give proper credit to other people's work, and to retribute your usage of everyone's contribution by simply contributing back. It's not too much to ask, is it?

    • Please stop trying to divide this community. In particular, ZXDB is a community effort, it will always remain open and freely available for everyone to use it honestly, including new WoS. An attempt from new WoS to compete, by taking an older version of ZXDB and then trying to update it independently, will prove nothing and it's just plain stupid. Instead, it would benefit everyone to work together and join efforts to improve it further. For instance, years ago Gerard was organizing magazines for WoS, so if he has anything to contribute with ZXDB now, it will be more than welcome. Likewise, the What's New page at new WoS shows that new WoS online update tool is getting used almost exclusively to add RZX files one-by-one, but a simple upgrade to latest ZXDB would already add everything automatically at once. However, if you don't want to cooperate, fine. As Lee Fogarty stated several times, data from WoS is "open source and fully available", thus from now on, I won't even bother to check if someone sees an useful update at new WoS and decides to send it to ZXDB too. Anyway, my offer for cooperation still stands, you know how to contact me publicly or privately, I'm still available, willing to help and make things work out. Either cooperate or leave it alone, it's your call. Just please cut the crap.

    Now I have no intention to post about this problem again. Please don't drag me back.
    Post edited by Einar Saukas on
    Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
  • I post rather rarely, but now I have to say that I agree with Einar and I fully understand why he does what he does. It's a shame that all these problems happens. WoS should have started using ZXDB openly right from the beginning of ZXDB's existence.
  • It is very admirable from Einar that although he could well be understood if he decided to avoid this issue and the WOS forum like bubonic plague, he still offers himself to settle everything and move on. I really do not understand the point of these continuous (and gratuitous to boot) attacks against him. Especially after he repeatedly produced such a solid base to support his affirmations.

    I am sure it would be a benefit for the whole community to try to act together and opnely share information and work instead of indulging into unfounded claims, witch hunt, and the suchlike.
  • How hard is it to credit Einar for using his ZXDB, and why avoid doing it anyway?
  • ... and I was dragged back to this crap here.
    Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
Sign In or Register to comment.