+3 disks and IPF format

edited December 2011 in Emulators
In order to avoid derailing the Spectaculator 7.5 release thread, I've created this thread for further discussion of the IPF format for +3 disks that was opened by the following new feature of Spectaculator:
- Spectrum +3 disk images in IPF format are now supported. IPF disk images will be available shortly. See http://www.softpres.org/ for more information on the IPF format.

There were a couple of comments I wanted to respond to:
You can use SamDisk 3 to convert IPFs to EDSK.
Emulator authors might be able to just call SamDisk to convert before load ?
VICE did that for TZXs calling 64TZXTAP before loading.

I'm not sure if that works under the license agreement, calling 3rd party plugins/utils ?

I think it is ungainly though it would theoretically work under the license agreement (for separate utilities rather than plugins) but only for places where Samdisk and the IPF libraries exist, i.e. Windows 2000 and newer as far as I can see. That doesn't help everyone else on everything from mobile phones to Macs.

I think it is really unfortunate that the end result of lots of effort in recovering software from secret encodings on disk is their "preservation" in a secret format that can only be access by secret source code. In my opinion it is self-evident that we could be left with that software locked up and inaccessible in many years, just look at how the BBC nearly lost the Doomsday Project http://en.wikipedia.org/wiki/BBC_Domesday_Project. Librarians the world over seem to be well aware of the need to preserve content in open formats rather than proprietary ones.

I agree with the comments that the emergence of IPF as a disk distribution format is a retrograde step in software preservation.
Dunny wrote: »
I'm sure that when IPF +3 Disk images start to appear for download, we'll be able to start reverse-engineering the format so that it can be opened up and included in free OSS libraries.

I hope that is true. The SPF guy who came into WoS when this was mentioned during the TZX wars was adamant that the format was a small part of the picture and that the library works very differently to other emulation. I suspect that implies a much larger reverse-engineering effort over the whole IPF-handling dll rather than being able to concentrate on the IPF file alone.
Post edited by Fred on

Comments

  • edited February 2011
    ... For softare preservation, you ideally want to dump to draft using Kyroflux interface, as when dumping in draft, it preserves all of the magnetic flux on media at the lowest level - in a nutshell it's the closest you will get to preserving a disk (which could even be analysed at phorensic level). The downside is no emulators support the draft format, and if you want to rewrite stuff back to disk then write support via Kyroflux they are still working on. Awesome piece of kit though for software preservationists :)

    Draft format for dumping disks is an open format, so it just up to the emulator authors out there to code support for it into their emulators.

    Kyroflux interface:
    http://www.kryoflux.com

    More in depth info here:

    http://www.softpres.org/glossary:KryoFlux
  • edited February 2011
    I don't get it. Why do we need the IPF format for preservation of +3 disks again?

    I gather that it's some sort of universal disk format but does it offer any advantage over current disk formats that are already being supported by a majority of emulators? Also, it doesn't seem to bring TR DOS disks under the umbrella at the moment so in effect it just seems to replace the .dsk format disks and that's about it as far as the speccy goes.
  • edited February 2011
    Arjun wrote: »
    I don't get it. Why do we need the IPF format for preservation of +3 disks again?

    I gather that it's some sort of universal disk format but does it offer any advantage over current disk formats that are already being supported by a majority of emulators? Also, it doesn't seem to bring TR DOS disks under the umbrella at the moment so in effect it just seems to replace the .dsk format disks and that's about it as far as the speccy goes.

    On the one hand, there's no real alternative to .ipf if you want to properly preserve disks (at least as far as the Amiga goes - so we're told by the IPF folks).

    On the other hand, it's closed down by the people who developed it which is why it's not been used for Speccy development - even if our emulators are not all open-source, we've all been very forthcoming with info to help others develop their software. The IPF (or CAPS) folks are not "open" for reasons never really made clear.

    Up until now, most emulator authors have resisted using IPF despite pressure from the CAPS supporters to add it - and we decided against it as a moral/ethical thing (all apart from Woody, that is, he has no morals and doesn't support IPF through sheer laziness ;) ).

    Not sure why Jon decided to support IPF as e-dsk can be levered to support all the disks in the +3's library. It's not pretty, but there was really no need to bow down to the IPF/CAPS nazis. Still, it's his emulator and his revenue stream so I guess he gets to decide :-)

    And of course, as with TZX block $19, if any archive maintainers do decide to store their disk images in IPF format then they should bear in mind that only one speccy emulator supports them, and that only under windows.

    D.
  • edited February 2011
    Arjun wrote: »
    I don't get it. Why do we need the IPF format for preservation of +3 disks again?

    I gather that it's some sort of universal disk format but does it offer any advantage over current disk formats that are already being supported by a majority of emulators? Also, it doesn't seem to bring TR DOS disks under the umbrella at the moment so in effect it just seems to replace the .dsk format disks and that's about it as far as the speccy goes.

    Hi Arjun,

    IPF would be in addition to the SDP project and EDSK, but won't impact WoS as IPF will not be available from here. It's just another option should you want 100% authentic disks check and verified by "professionals" (for want of a better description). With SDP and EDSK, while I can be about 99% sure that the disk is good, I have no way to authenticate whether the disk has or has not been modified in anyway. The SPS guys can without reasonable doubt.

    It isn't meant to replace EDSK or SDP or SPA2 for that matter, it's just another option which you can choose to use or not, not a problem.

    Jon was so kind to help us have an emulator to test things with, and a brilliant job it does.

    There's going to be an open DRAFT format, so for the many who are buying Kryoflux units (like myself ;-) ) can dump disks at the finest resolution and do what they like with this. Make their own IPF-like format or whatever they choose. IPFs are really a result of the riggerous testing the guys at SPS do to ensure any IPF releases is 100% authentic.
    The DRAFT will be akin to making a high quality WAV of a tape. This might be a crude way of putting it, but I think that's a similar description of it.

    There's a number of Amstrad CPC disks that need extra extra extentions to the EDSK format, some were defined by Kevin Thacker and some by Simon Owen.

    Regarding other disks formats. I've dumped a number of Opus Discovery disks, but they've not yet been processed. There seems to be an issue with some of the Boots compilations that have a bad sector, which could be copy protection. The OPD format is too simple to store these but I have EDSK dumps of them.

    I'm not sure about TRD disks, but you can dump them with the Kryo hardware if you wanted to. Were there any commercial disks released on TRD/beta disc ?

    Anyway, don't sweat it, there's no plans to put IPF in the place of anything we currently have. EDSK can store all the disks we've dumped for the speccy so far.

    Cheers

    Andy
  • edited February 2011
    IPFs are really a result of the riggerous testing the guys at SPS do to ensure any IPF releases is 100% authentic.

    Why does that require the format to be closed?

    It seems to me you've been drinking a little bit too much of the Kool-Aid again without applying a critical eye to what you're being told.
  • edited February 2011
    It seems to me you've been drinking a little bit too much of the Kool-Aid again without applying a critical eye to what you're being told.

    Indeed. I can vividly remember trying to get through to Mr Barker that MakeTZX was actively corrupting TZX files (by removing decryption/protection tones which RealSpec didn't need), and having a devil of a job doing so.

    Andy, do you really believe what the CAPS/IPF lads tell you? Really?

    D.
  • edited February 2011
    Hercules wrote: »
    ... For softare preservation, you ideally want to dump to draft using Kyroflux interface, as when dumping in draft, it preserves all of the magnetic flux on media at the lowest level - in a nutshell it's the closest you will get to preserving a disk (which could even be analysed at phorensic level). The downside is no emulators support the draft format, and if you want to rewrite stuff back to disk then write support via Kyroflux they are still working on. Awesome piece of kit though for software preservationists :)

    Draft format for dumping disks is an open format, so it just up to the emulator authors out there to code support for it into their emulators.

    I don't think this is intended to be a usable format equivalent to DSK etc. According to http://www.softpres.org/glossary:draft
    It may work with emulators, but functionality is not guaranteed and may be unreliable. Some types of protections will not work.

    I guess that you need to detect the protection in use etc. to be able to use a Speedlock or similar protected disk.

    I also couldn't find any specification for the format on softpres.org so I think it is an idea rather than a reality. It would appear to be akin to the difference between a WAV file (unstructured and as found on a read) and a TZX file (structured "analysed" interpretation of the data) with DUMP being the former and IPF the latter (loosely speaking).

    I don't think it is a particularly useful format for preservation for those reasons. I think that IPF is also a bad format for preservation as it is closed, a point that seems to be acknowledged by SPS, e.g. see http://www.softpres.org/faq:general:open_source:
    Will you release the file format and/or source code for handling IPF files in the future?

    Yes, absolutely. To not do would be rather flawed preservation.

    This statement was made in 2006 according to the datestamp on that page. In June last year Kieron from SPS says the following here http://www.classiccmp.org/pipermail/cctech/2010-June/115481.html:
    > Mmm. Not releasing the IPF file format info is pretty inexcusable.

    You are of course absolutely right in this regard. We have known since the beginning that it was a pretty ridiculous state of affairs for a image format intended for preservation (we have publically said this in multiple places elsewhere). Fortunately the reasons behind it have disappeared over the years, the remaining thing to do was multi-system support (now done, as of last week). It now basically comes down to finding the time to sort it out. We wanted to do a few other things first (like being able to show that an image really comes from us), but we have been discussing the subject internally recently, and we will probably defer that. It's high time it was available. Feel free to keep complaining until we do it. :)

    And there seems to be no further information to share since then. If and when this is opened up, it sounds like this would be an excellent tool but until then it seems to me that the existing disk formats are better than IPF for preservation in every material way (i.e. they exist in a form where the required handling can be reimplemented from scratch from documentation alone and as far as I know have produced operational dumps of all protected games?).
  • edited February 2011
    Dunny wrote: »
    Not sure why Jon decided to support IPF as e-dsk can be levered to support all the disks in the +3's library. It's not pretty, but there was really no need to bow down to the IPF/CAPS nazis. Still, it's his emulator and his revenue stream so I guess he gets to decide :-)

    Jon was just kind enough to offer support for us to test the images. It was a selfless act, I don't think he expects it would boost sales or anything. Much like you guys who have helped in many areas over the years.
    And of course, as with TZX block $19, if any archive maintainers do decide to store their disk images in IPF format then they should bear in mind that only one speccy emulator supports them, and that only under windows.

    D.

    A note would be put at the top of the page, same should be noted for v1.20 TZXs.
    They can be converted with SamDisk anyway. No big deal, they'd only be there IF someone was interested in having them. I'm just happy that files have come into fruitition and we're able to test/convert them and have the knowlegde that they're athentic beyond doubt.
  • edited February 2011
    Why does that require the format to be closed?

    It seems to me you've been drinking a little bit too much of the Kool-Aid again without applying a critical eye to what you're being told.

    To be honest I probably do. I like to see the best in everyone, which is why I respect you too ;-).

    I'm just a tape/disk monkey push over, and proud of it.
  • edited February 2011
    Hi Andy,
    IPF would be in addition to the SDP project and EDSK, but won't impact WoS as IPF will not be available from here. It's just another option should you want 100% authentic disks check and verified by "professionals" (for want of a better description). With SDP and EDSK, while I can be about 99% sure that the disk is good, I have no way to authenticate whether the disk has or has not been modified in anyway. The SPS guys can without reasonable doubt.

    So would you be able to update the SDP/EDSK images of the +3 software on WoS based on the IPF images from SPS using Samdisk 3? Do we know of any images that are different based on this authentication?

    That way the EDSK-using community would get the benefit of the verification until such a time as SPS get around to following up on their promises to open the format.

    Will SDP/WoS be distributing IPF files?
  • edited February 2011
    Dunny wrote: »
    Indeed. I can vividly remember trying to get through to Mr Barker that MakeTZX was actively corrupting TZX files (by removing decryption/protection tones which RealSpec didn't need), and having a devil of a job doing so.

    Andy, do you really believe what the CAPS/IPF lads tell you? Really?

    D.

    I have no reason not, to. However my SDP and SPS dumping run side by side, so I don't mind. I'm happy to support and provide images to them.

    If I didn't put trust in anyone, we'd never move anywhere or get anything done, stalemate, stagnant. I wanna keep moving forwards. I'll probably support anyone who thrown me a preservation bone, cos that's all I wanna do, provide things for people.

    Anyway, cool beans, I'm happy.....
  • edited February 2011
    Fred wrote: »
    Hi Andy,

    So would you be able to update the SDP/EDSK images of the +3 software on WoS based on the IPF images from SPS using Samdisk 3? Do we know of any images that are different based on this authentication?

    That way the EDSK-using community would get the benefit of the verification until such a time as SPS get around to following up on their promises to open the format.

    Will SDP/WoS be distributing IPF files?


    Regarding the first paragrah, yes I could update any image that had been modified (fixed) by removing bad sectors in known/assumed unused space with Disk Image Manager. There's only a few. If I had a good IPF, I can convert this to ESK compare and okay for release. I haven't converted the 130 images I currenly have and compared them yet. I looked at a few and they seemed the same with respect to sector visual check.

    WoS will NOT be distributing IPF files, these may be on the Vault if Steve decides.

    Cheers

    Andy
  • edited February 2011
    Thanks for the info, I've a couple of supplementary questions now :)
    Regarding the first paragrah, yes I could update any image that had been modified (fixed) by removing bad sectors in known/assumed unused space with Disk Image Manager. There's only a few. If I had a good IPF, I can convert this to ESK compare and okay for release. I haven't converted the 130 images I currenly have and compared them yet. I looked at a few and they seemed the same with respect to sector visual check.

    That's good and I think would be appreciated when you get the time. It is also reassuring that the disks checked so far have been of a similar quality to the IPFs from the "professionals" (that is the data seems valid even if there is some junk cluttering the images). I think everyone would like images to be as good as possible.

    I am a bit confused about the method though, why wouldn't you just use Samdisk to bulk convert as you were suggesting above? The manual approach you mention sounds more time-consuming than a batch job.
    WoS will NOT be distributing IPF files, these may be on the Vault if Steve decides.

    I'm curious why WoS won't be doing so? Is to do with extending the database for example?
  • edited February 2011
    Hmm on second reading I may be misinterpreting - I think you are saying you would use Samdisk but would be checking the produced file against the existing EDSK before releasing? I think you threw me off with the comment about removing the bad sectors manually?
  • edited February 2011
    Hi Andy,

    One more thing - despite the disagreements that crop up on file format details, I did want to say that I greatly appreciate the effort you and the rest of the preservation team put into making the best images of the Speccy software archive that you can. Thanks very much for all of that :)
  • edited February 2011
    Hi Fred,

    When I say quality I mean that the data appears to be the same on a by eye visual level.
    I've not compared them datawise like I do for TZXs with Tapir for instance. Just looking
    at the disk layout structure in Disk Image Manager to see if they look the same.

    Really it's the copy protected disks I'd need to look in more detail as standard disks would
    just dispense of anything extra that an IPF may hold. It's more about authentication down to the duplication method that I can't check, and is probably beyond me.

    I guess I could just go and replace all the currently available IPFs->EDSKs and replace what's been done for SDP. Though if the resultant image was the same, then it's perhaps
    overkill to replace them for little gain in that respect. I do need to check some images, but I'm not even sure if I have IPFs of those. Fortunately Kryo dumps submitted to SPS (Spectrum and Amstrad) and more people able to dump means we'll get more dumped.

    Looks like I've got some homework to do then ;-).
  • edited February 2011
    Fred wrote: »
    Hmm on second reading I may be misinterpreting - I think you are saying you would use Samdisk but would be checking the produced file against the existing EDSK before releasing? I think you threw me off with the comment about removing the bad sectors manually?

    No worries mate, I do babble on, and say too much most of the time ;-).

    I'd only really check current EDSKs against IPF->EDSKs if I suspected they needed replacing due to out of date weak sectors/copy protection, or if the disk had been modifed by me removing bad sectors in unused space. Much like me stripping zeros at the end of a TZX block.

    I suspect you'll find that an IPF->EDSK conversion would compre similar if not equal to the EDSKs already available. It's just that I know the IPFs are good and could use them to check against the SDP release if someone raised an issue with something working or any corruption that shouldn't be there.

    You may then ask what the point of replacing any EDSKs with IPF->EDSK would be. Well for SDP it's just an extra checking mechanism if we find a suspect disk that needs re-addressing.

    Many thanks

    Andy
  • edited February 2011
    Fred wrote: »
    Hi Andy,

    One more thing - despite the disagreements that crop up on file format details, I did want to say that I greatly appreciate the effort you and the rest of the preservation team put into making the best images of the Speccy software archive that you can. Thanks very much for all of that :)

    Many thanks Fred, much appreciated.

    That's fine mate, disagreements and opinions are always welcome, helps us drive our desire to keep the scene alive.

    I think I just like to please too many people, but that's not a bad thing is it ?

    Best wishes

    Andy
  • edited February 2011
    You may then ask what the point of replacing any EDSKs with IPF->EDSK would be. Well for SDP it's just an extra checking mechanism if we find a suspect disk that needs re-addressing.

    So, from a preservation point of view I can see why IPF could be of use, but from an emulation point of view I suppose the rest of us needn't lose sleep over it. We'll just leave it to you to provide us with quality EDSKs in the archive. ;)
  • edited February 2011
    Indeed that's right Arjun, no need to loose sleep as we'll still have EDSK, the plan was never to replace anything, just to facilitate an addition check.

    We can go back to sleep now ;-).
  • edited February 2011
    Fred wrote: »
    I'm curious why WoS won't be doing so? Is to do with extending the database for example?

    Adding a filetype is easy. :-)

    However, my reluctance is exactly what has been brought forward earlier in the thread: closed formats are ultimately bad for preservation purposes. As soon as the developer calls it a day, all files will be worthless forever.

    As I'd rather wait for the format to be opened up, I'll refrain from supporting (and thus encouraging) it for the time being.

    Another factor is that this is "just" another disk format and I'd rather support as few formats as possible to get the best experience from the existing emulators.
  • edited February 2011
    The reason I don't like it is because just lately, having started developing for architectures and platforms non-x86/Windows I've been convinced of the benefits of truly open standards, and should I decide to port (or write) an emulator for my Pandora, I'll not be able to support IPF files on it - there is little chance that the libraries necessary will be available to me to use under ARM-Linux.

    D.
  • edited February 2011
    There's going to be an open DRAFT format, so for the many who are buying Kryoflux units (like myself ;-) ) can dump disks at the finest resolution and do what they like with this. Make their own IPF-like format or whatever they choose. IPFs are really a result of the riggerous testing the guys at SPS do to ensure any IPF releases is 100% authentic.
    The DRAFT will be akin to making a high quality WAV of a tape. This might be a crude way of putting it, but I think that's a similar description of it.

    Draft is the way to go... rather than a sector by sector copy of a disk, DRAFT is as precise as you'll get... it will even record the magnetic strength of the flux of a disk according to the Kyroflux guys, which for software preservationists is as close you will get to a exact duplicate of a floppy.

    Hats off to Dunny regarding the IPF format; I enjoyed reading his post on the Kyroflux forums, and I have to agree - if their not going to release the specifications of how IPF's work, then reverse engineer it. Formats for emulators should be open for all so the format(s) can be supported. Reverse engineering is also a good way to piss of the developers, but if it gets the job done, then go for it buddy. LOL :D

    See:

    http://forum.kryoflux.com/viewtopic.php?f=2&t=136

    Dunny's post below read:-
    Dunny:

    4. If the format is not opened and you insist I use the library then I shall reverse engineer the format and publish my work. I have no need to concern myself over a complete implementation - just enough to read and write ZXSpectrum disk images. I suspect you'd prefer that disks created in IPF format adhere to your spec, not an incomplete one. There has been discussion amongst certain folks in the Speccy scene, and there are more than just me interested in reverse engineering the format. This is likely going to be done by comparing .ipf files with e-dsk files, though I myself am certainly not above disassembling your library to see how it works ("clean room", if you will).
  • edited February 2011
    Hercules wrote: »
    Draft is the way to go... rather than a sector by sector copy of a disk, DRAFT is as precise as you'll get... it will even record the magnetic strength of the flux of a disk according to the Kyroflux guys, which for software preservationists is as close you will get to a exact duplicate of a floppy.

    I'm really not sure that is the case. DRAFT of course does not exist yet so there is an air of speculation about what it will or will not be or if it will ever exist, but I don't think that the SPS people agree with your suggestion that it is supposed to be a usable format for emulators, as I quote above, according to http://www.softpres.org/glossary:draft
    It may work with emulators, but functionality is not guaranteed and may be unreliable. Some types of protections will not work.

    It doesn't sound like a good format to use directly, and it isn't obvious that it sufficiently describes a disk to support protected disks (perhaps including Spectrum software).
  • oboobo
    edited February 2011
    DRAFT isn't really designed for end-users ? it's a high-resolution scan needing further processing to be of any use. The files will be huge, and if opened by an emulator it'd probably take 5 seconds or so to turn it into something most FDC emulations can use. DRAFT is like a 4800dpi scan of a letter, whereas traditional sector-based formats are the OCR output from processing it. It's a single interpretation of the data, but it's much more compact and convenient to use.

    IPF pretty much gives the best of both. It's a re-mastered and validated version of the original, retaining all the detail of the original disk but without the huge file size. IPFs generated from the same disk are exactly the same each time, whereas DRAFT dumps will always be different. The weakest link in the chain will then be the emulator FDC code, as you'll need a cycle-accurate emulation to take advantage of the full IPF detail. The level supported by most Spectrum emulators should still be good enough to handle all +3 disks, fortunately.

    Progress is being made on opening up the format and/or the library, so hopefully a solution compatible with everyone will be worked out soon.
  • edited February 2011
    What I found the most interesting bit in Dunny's comment was:

    "...when I go back to developing ZXSpin later this year".

    Surely, sweet music to the ears of Spin users! ;)
  • edited February 2011
    Arjun wrote: »
    What I found the most interesting bit in Dunny's comment was:

    "...when I go back to developing ZXSpin later this year".

    Surely, sweet music to the ears of Spin users! ;)
    And bad news for the rest of us Speccy emulator developers...
    I wanna tell you a story 'bout a woman I know...
  • edited February 2011
    mheide wrote: »
    As I'd rather wait for the format to be opened up, I'll refrain from supporting (and thus encouraging) it for the time being.

    This shouldn't take so long maybe:

    http://www.softpres.org/news:2011-02-09

    the format will be open, sooner or later..
  • edited December 2011
    mheide wrote: »
    Adding a filetype is easy. :-)

    However, my reluctance is exactly what has been brought forward earlier in the thread: closed formats are ultimately bad for preservation purposes. As soon as the developer calls it a day, all files will be worthless forever.

    As I'd rather wait for the format to be opened up, I'll refrain from supporting (and thus encouraging) it for the time being.

    Another factor is that this is "just" another disk format and I'd rather support as few formats as possible to get the best experience from the existing emulators.

    Is this the answer? http://forum.kryoflux.com/viewtopic.php?f=2&t=265 the source for IPF decoding has been released, thus making the IPF format open? (ducks from incoming flames and thread necromancy but feel this is important)
Sign In or Register to comment.