New DB

edited July 6 in Updates
Ok, time to clear this up. After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. Other have previously said they had the old database anyway. What they actually had was a very small selection exported to a CSV file from the main database. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

Having said that, like I said, there will never be an agreement. I will however say that the WoS admins are very good at telling me when I am wrong - and in this case all have agreed and stated over the last couple of years that my opinion here isn't wrong. Everyone in the admin group were working (as far as we were concerned) on Infoseek. I should also point out here that our "secret" facebook group is the official WoS fb and currently has over 5,000 members.

So, moving forward....

Having spent so much time with the data, it was clear that there were issues with the database. Issues that were so numerous, the decision was made to rebuild it, using the initial titles as a starting point - people will know this from the old CSV download - there's a lot more to it than just that one file, but it was a starting point. We've posted numerous updates over the last 2yrs, detailing the work we are doing, and where we are at. To give you some idea, every single title and release has been manually checked. This has led to tens of thousands of changes. Not just checking titles - this has involved opening and checking every inlay and scan on WoS, commenting on price differences, advertised versions, etc.. It's been a mammoth task.

As we neared the end of this, I was also putting together the new screens. It was intended to put this live at the end of July. In preparation for new scans (around 300gb), I purchased a higher spec server. Don't forget - we are an archive. It's a lot different to just linking to archive.org and hoping it stays up. This means we currently have 630gb files on the server. Some - such as the new inlays - aren't available yet.

The server move proved to be as bad as the last time, with the old cgi files not playing well. These scripts are almost 25yrs old. The only ones remaining that hadn't been moved to the new website were the infoseek related ones. So, one evening I switched the new pages on. I made no secret that it wasn't the database we will be using, as it is still being processed (fingers crossed this week). I have said many time here, and on the facebook group that this data isn't accurate and will not be used when our new database is ready to go live. However, the way the WoS database is constructed means that we can take change requests - not all can be actioned until we have the new set of data, but a lot can be. WoS has 91 tables in the database.

For an example of how much has gone into this, the old database has 28,699 titles & versions listed. This currently stands at 33,014 - and we haven't completed compilations or mag tapes yet. This is still phase 1, as we also have thousands of new scans to go through - they will be processed after we have finished compilations. This won't mean much to the average user - who cares that there are 5 versions of Licence to Kill and not the 3 currently listed? However, for the likes of collectors, this is important information.

The Infoseek engine currently has 308,807 items indexed, plus over 300,000 indexed, downloadable files. This isn't restricted to software - we have indexed every magazine page separately - something that the old WoS db didn't do. Because of the indexing, it has improved the infoseek results.

As for the data - I appreciate people are waiting for the advanced search to come online, but as it stands at the moment, with pages and sections missing, we still have far more information available to download than ever before - and based on feedback, that can only increase. The API is comprehensive and provides a lot more than the older one, and is also still being built upon. We have built this data up with accuracy at the front at all times. I am sure there will be some mistakes in there - it's to be expected.

So, what now?

The old database is considered legacy. As of today, the old database can be taken and used at will. However, I would like all mention of myself & WoS removing. It is provided "as is" without any guarantee of accuracy or quality.

The correct database will be loaded in the next few days.
Post edited by Lee Fogarty on
My test signature
Thanked by 2polomint mik3d3nch

Comments

  • edited July 7
    It shouldn’t always fall on the same person to refute this sort of rubbish. There are plenty of others who could do it, and probably do it better than me, but I’ll take a turn.
    Ok, time to clear this up.

    Great. Finally, we’ve been waiting nearly three weeks for a response.
    After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

    Hang on… the creator of ZXDB has made 22 posts about this in your own forum over the last 19 days:

    https://worldofspectrum.org/forums/discussion/57777/new-wos-and-zxdb/p1

    He has posted detailed and extensive evidence showing how you have taken your data from ZXDB, while falsely accusing him of stealing your data. He couldn’t PM you, as you have blocked him from sending PMs.

    He’s also posted all the same evidence at the Spectrum Computing forum, where you have an account.
    Firstly, there will never be an agreement as to what happened with the old data.

    Why ever not? Surely two reasonable adults could discuss the facts, and work out exactly what has happened. Or are you trying to avoid the facts and turn this into a game of “he-said-she-said”?
    I say it was take in its entirety without permission, Einar says otherwise.

    It is a matter of public record that you gave permission for Einar to use WoS data to start ZXDB. It is recorded in this thread:

    https://worldofspectrum.org/forums/discussion/52892/had-enough/p1

    To summarise the thread, you decided to quit WoS and looked for someone else to take over. Einar offered to create ZXDB to help. You agreed. WoS admins even then praised Einar for the work he did in creating ZXDB.

    The data wasn’t stolen. You and Rich Chandler keep repeating this accusation, but you never offer any credible evidence for it and you never offer any refutation of the abundant evidence showing that the data was not stolen.
    Other have previously said they had the old database anyway. What they actually had was a very small selection exported to a CSV file from the main database. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

    Einar has already addressed this comprehensively and in more detail than I could:

    https://worldofspectrum.org/forums/discussion/57742/is-ftp-wos-available/p1
    Having said that, like I said, there will never be an agreement.

    The “he-said-she-said” stuff again. We aren’t discussing the existence of God. We are discussing some pretty simply facts. This issue really can be settled by looking at the facts.
    I will however say that the WoS admins are very good at telling me when I am wrong - and in this case all have agreed and stated over the last couple of years that my opinion here isn't wrong. Everyone in the admin group were working (as far as we were concerned) on Infoseek.

    Are you seriously arguing “my close mates agree with me, so I must be right”???
    I should also point out here that our "secret" facebook group is the official WoS fb and currently has over 5,000 members.

    ….which Einar cannot see or participate in, so it is secret from him.
    So, moving forward....

    Having spent so much time with the data, it was clear that there were issues with the database. Issues that were so numerous, the decision was made to rebuild it, using the initial titles as a starting point - people will know this from the old CSV download - there's a lot more to it than just that one file, but it was a starting point. We've posted numerous updates over the last 2yrs, detailing the work we are doing, and where we are at. To give you some idea, every single title and release has been manually checked. This has led to tens of thousands of changes. Not just checking titles - this has involved opening and checking every inlay and scan on WoS, commenting on price differences, advertised versions, etc.. It's been a mammoth task.

    As we neared the end of this, I was also putting together the new screens. It was intended to put this live at the end of July. In preparation for new scans (around 300gb), I purchased a higher spec server. Don't forget - we are an archive. It's a lot different to just linking to archive.org and hoping it stays up. This means we currently have 630gb files on the server. Some - such as the new inlays - aren't available yet.

    The server move proved to be as bad as the last time, with the old cgi files not playing well. These scripts are almost 25yrs old. The only ones remaining that hadn't been moved to the new website were the infoseek related ones. So, one evening I switched the new pages on. I made no secret that it wasn't the database we will be using, as it is still being processed (fingers crossed this week). I have said many time here, and on the facebook group that this data isn't accurate and will not be used when our new database is ready to go live. However, the way the WoS database is constructed means that we can take change requests - not all can be actioned until we have the new set of data, but a lot can be. WoS has 91 tables in the database.

    For an example of how much has gone into this, the old database has 28,699 titles & versions listed. This currently stands at 33,014 - and we haven't completed compilations or mag tapes yet. This is still phase 1, as we also have thousands of new scans to go through - they will be processed after we have finished compilations. This won't mean much to the average user - who cares that there are 5 versions of Licence to Kill and not the 3 currently listed? However, for the likes of collectors, this is important information.

    The Infoseek engine currently has 308,807 items indexed, plus over 300,000 indexed, downloadable files. This isn't restricted to software - we have indexed every magazine page separately - something that the old WoS db didn't do. Because of the indexing, it has improved the infoseek results.

    As for the data - I appreciate people are waiting for the advanced search to come online, but as it stands at the moment, with pages and sections missing, we still have far more information available to download than ever before - and based on feedback, that can only increase. The API is comprehensive and provides a lot more than the older one, and is also still being built upon. We have built this data up with accuracy at the front at all times. I am sure there will be some mistakes in there - it's to be expected.

    So, what now?

    The old database is considered legacy. As of today, the old database can be taken and used at will. However, I would like all mention of myself & WoS removing. It is provided "as is" without any guarantee of accuracy or quality.

    The correct database will be loaded in the next few days.

    And….

    And….

    Is that it?

    You’ve written ten paragraphs and not addressed a single one of Einar’s points. Worse, you’ve not even mentioned ZXDB once!

    Why does the database you are using exactly match an out-of-date version of ZXDB? Why has it put through verbatim the same hundreds of changes as that old version of ZXDB? Why does it contain multiple links and references to ZXDB? Why has New WoS suddenly started to misspell ZX Spectrum as “ZX-Spectrum” just as a certain Brazilian has for years?

    If you intended to use ZXDB for this version, why not say it? And why did you feel the need to keep removing evidence showing you were using ZXDB? If it wasn’t the “real” database, what was the point?

    Einar has said repeatedly that he has no objection to WoS using ZXDB. What is unreasonable is that you and Rich Chandler repeatedly make false allegations that he stole the data, while you do exactly the things you accuse him of. What is unreasonable is that you denigrate ZXDB while it seems to be the source of your improvements.

    Please stop engaging in this absurd, divisive behaviour. If you want to use ZXDB, use it properly and honestly. If you don’t, just do your own thing. And either way, stop making false accusations.


    Post edited by Rorthron on
    spectrumcomputing.co.uk/forums
    Thanked by 1zxbruno
  • Ok, time to clear this up.
    Nothing is cleared up from that post.


    Rorthron wrote: »
    You’ve written ten paragraphs and not addressed a single one of Einar’s points. Worse, you’ve not even mentioned ZXDB once!

    Why does the database you are using exactly match an out-of-date version of ZXDB? Why has it put through verbatim the same hundreds of changes as that old version of ZXDB? Why does it contain multiple links and references to ZXDB? Why has New WoS suddenly started to misspell ZX Spectrum as “ZX-Spectrum” just as a certain Brazilian has for years?

    If you intended to use ZXDB for this version, why not say it? And why did you feel the need to keep removing evidence showing you were using ZXDB? If it wasn’t the “real” database, what was the point?

    Einar has said repeatedly that he has no objection to WoS using ZXDB. What is unreasonable is that you and Rich Chandler repeatedly make false allegations that he stole the data, while you do exactly the things you accuse him of. What is unreasonable is that you denigrate ZXDB while it seems to be the source of your improvements.

    Please stop engaging in this absurd, divisive behaviour. If you want to use ZXDB, use it properly and honestly. If you don’t, just do your own thing. And either way, stop making false accusations.
    Well said.

    Oh, no. Every time you turn up something monumental and terrible happens.
    I don’t think I have the stomach for it.
    --Raziel (Legend of Kain: Soul Reaver 2)

    https://www.youtube.com/user/VincentTSFP
  • VincentAC wrote: »
    Nothing is cleared up from that post.

    Exactly.

    A huge wall of text which doesn't address anything.
  • For balance, since I had moan at Einar, I'll agree with the posters above.

    Einar has gone on and on with walls of text making claims, and the above doesn't clear any of that, so it may as well not have been posted. May as well just have said 'we disagree' and left it at that.

    I'm largely indifferent to it, I'm not really bothered who did what, but it is causing problems for a very vocal minority.
  • edited July 8
    FOR HEAVEN'S SAKE! WE ARE RUNNING IN CIRCLES AGAIN!!!

    Ok, time to clear this up.

    Except you didn't. I have proved beyond any doubt that new WoS "stole" ZXDB, despite you explicitly denying it and accusing me of the opposite. You didn't even try to explain anything I wrote.

    Instead, you are just following the Trumpesque principle of "never give up no matter how much evidence they have against you, just deny everything then try a different excuse".

    After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

    "Your honour, the robbed bank should have contacted me first before calling the police!"

    Are you seriously suggesting that this problem was caused because I didn't try hard enough to discuss ZXDB with you? I wrote over 5,500 posts in this forum, about 1,000 of them about ZXDB. There are entire threads that I started in this forum exclusively to attempt discussing ZXDB with you. My invitation to join the WoS admin group on Facebook in February 2017 was a direct response to my PM to both you and Richard Chandler, trying to sort this out privately.

    Except this time I couldn't send you a PM, because you disabled my PMs (everyone can still see "Roles: No PM" in my profile) and you never answered the reason you did it to me. Even so, this time everything I wrote was posted first in your own forum for the last 3 weeks. Wasn't it enough to "make contact" with you? Then I guess you don't visit your own site very often.

    Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise.

    "Your honour, evidences show I'm guilty but I won't confess, therefore you cannot condemn me since we will never agree."

    Except it's too late. Let's see:

    Attempting to rewrite history doesn't really work unless you have your own trustworthy portable time travel device.

    What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

    It seems you have fallen into a temporal paradox and got stuck in a time loop.

    First of all, you have to decide what exactly your "opinion" means:
    • Are you referring to the time I joined the WoS admin group on Facebook in February 2017? I spent 3 months asking for your new database model (so I could import data from ZXDB for you), you kept telling me it wasn't ready and providing new excuses, until I finally gave up in May 2017. In your parallel universe, it's when I grabbed a new WoS database, travelled back in time and released everything in August 2016.

    I will propose something you never considered so far: proof! In the private WoS group you wrote "There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him". I'm glad you have evidence of my wrongdoings from a parallel universe (presumably my mirror version is evil). Now you only have to teleport this evidence to our own universe, so you can publicly release our private WoS admin group discussions about ZXDB. Don't worry about privacy, you already had my approval to release from years ago.

    Having said that, like I said, there will never be an agreement. I will however say that the WoS admins are very good at telling me when I am wrong - and in this case all have agreed and stated over the last couple of years that my opinion here isn't wrong. Everyone in the admin group were working (as far as we were concerned) on Infoseek.

    "Your honour, my friends believe I'm innocent therefore I cannot be guilty!"

    Seriously?

    I should also point out here that our "secret" facebook group is the official WoS fb and currently has over 5,000 members.

    "Your honour, I made false accusations but they were not secret and I told them to thousands of people!

    I never said WoS group was "secret". I wrote "private WoS group at Facebook that I cannot access". Guess what? It's still private and I still cannot access.

    Posting false claims about ZXDB behind my back to "over 5,000 people" in a private group doesn't make it any better.

    So, moving forward [...] So, what now?

    "Your honour, I was caught driving a stolen car but I will be driving a different car next week"

    Really?

    The old database is considered legacy. As of today, the old database can be taken and used at will. However, I would like all mention of myself & WoS removing. It is provided "as is" without any guarantee of accuracy or quality.

    OK, I just uploaded the "old database" here.

    DISCLAIMER: These files are provided "as is" without any guarantee of accuracy or quality.

    These are the famous "Martijn's old WoS internal files" that I mentioned several times over the years, but never shared before. There are 8 packages, I named them according to the date when I received each file (from July 13th 2016 to August 3rd 2016, except "zines.dat" that was missing and I only received later on February 20th, 2017). If anyone wants to compare these files against ZXDB, I suggest downloading the ZXDB incremental scripts (from the same link provided at the end of my first post). Every single change in ZXDB is preserved in these scripts for auditing, so it's possible to see exactly how ZXDB evolved over time. The ZXDB files are organized as follows:
    • 002 - Scripts to populate enumeration tables.
    • 003 - Scripts to populate a list of magazine names.
    • 007 to 010 - These scripts contain other inconsistency fixes that my conversion system could not fix automatically, so I applied them myself afterwards. I have explained it in August 2016.
    • 011 - This is a script that added SPOT/SPEX content as separate tables. It's basically the same content that's available for download at SPOT/SPEX, except Gerard told me that Richard had already converted it to CSV/SQL, as I mentioned for instance in August 2016 and again in October 2016. [Although I don't recall which Richard so if anybody has this information, please let me know since I just realized his name is missing from ZXDB credits!] Gerard sent me these files on July 13th 2016 but I only added them to ZXDB later because my first priority was converting old WoS, afterwards I started working on SPOT/SPEX in September 2016.
    • 012 to 102 - The remaining scripts contain incremental changes to fix errors and inconsistencies, and add new information. Some SQL was generated manually, others produced from spreadsheet macros, and more recently using an editing tool called ZX-Ed that I created for this purpose.

    The organization of these files is still the same that I already explained in October 2016. Although in a certain parallel universe, everything was just a dream.
    Post edited by Einar Saukas on
    Creator of ZXDB, BIFROST/NIRVANA, ZX7/RCS, etc. I don't frequent this forum anymore, please look for me elsewhere.
  • edited July 8
    Einar - you're wasting your time. There's probably a maximum of three people involved who disagree with you, and they've gone suspiciously quiet. It's like arguing with my dog - pointless.

    I'd wager money that Lee won't reply, or if he does he'll ignore the obvious holes in his own arguments. And for the sake of what - a little bit of text somewhere in the site that says "thanks for ZXDB guys, we used it for a couple of weeks"? Given the very public history of ZXDB - detailled beautifully above - what is "the problem" here? There's no monetary gain, no-one got physically hurt - so it must just be blind ego, mustn't it? Please, Lee - tell us all why it matters so much.
    Post edited by KenD on
  • It seems you have fallen into a temporal paradox and got stuck in a time loop.
    gd.jpg

  • Ffs, lee. What gives dude. You used to do a lot of good around these parts :-(
    Just seems when a hole is dug, you like going deeper. Pull yourself out man, really, why do you enjoy being a pariah ?
    Damn shame :-(
Sign In or Register to comment.