MuCho multiple choise adventure engine, release 2 - little to no programming needed

Sol_HSA · October 2016

Release 2 of the MuCho multiple choise adventure (CYOA, choose your own adventure, gamebook) game engine.

http://iki.fi/sol/zip/mucho_02.zip

For those who just want to look at the compiled examples and manual: http://iki.fi/sol/temp/
More specifically, http://iki.fi/sol/temp/mucho.pdf for the pdf manual.

Changes:
- Completely redone documentation, some 35 pages in PDF form (also available in html, mobi and epub)
- Animated selector
- Custom fonts, dividers and selectors
- More solid look and feel, hiding work from user
- Bunch of custom fonts, dividers and selectors included in the kit
- More keys checked for input
- Little bits of polish here and there
- Lots of little bugs fixed

Hikaru · October 2016

Looking good so far!

I'd like to suggest making the animated cursor feature optional in case it's not (couldn't find it in the manual). These sorta things might look good at times, while at other times they can be seen as distracting, depending on the mood/story the author is trying to portray

Are you planning to support calling custom user routines in some way? Let's say AY/beeper music or animated cutscenes

Sol_HSA · October 2016

Hikaru wrote: »

I'd like to suggest making the animated cursor feature optional in case it's not (couldn't find it in the manual). These sorta things might look good at times, while at other times they can be seen as distracting, depending on the mood/story the author is trying to portray

It's not optional, but you can customize how it looks like. Putting a solid horizontal bar there would still be animated but you wouldn't see the animation since all animation frames look the same.

Hikaru wrote: »

Are you planning to support calling custom user routines in some way? Let's say AY/beeper music or animated cutscenes

I suppose I could let you uncompress raw code to the 4k buffer and then jump to it, but I'm pretty sure if you're capable enough to write code that could live there, you'd be more than capable enough to add that functionality yourself =)

Anyway, unless people actually DO things with this, I'm pretty much done on the engine side. Planning to make a game or two with it myself, we'll see what kinds of things I come up with during that.

MykeP · October 2016

Very interesting Sol_HSA.

A number of years ago I 'remade' classic CYOA book; The Dealy Shadow in HTML.

I'd planned to write a .NET application (named CYOdA with the little 'd' being 'digital') that would act as an Adventure Editor and export the flat HTML files ready for upload to your web server. This 'remake' was my manual trial run to determine the data structures I would need.

I also contacted the publisher at the time to see if they'd allow me to put up the demo on my website, however, I received no reply.

I think this knocked my confidence a bit and CYOdA became yet another name on my list of unfinished projects. ;)

Anyway my immediate thought when I saw this was; given the book was 118 pages (maybe 100 words on the longest page) with only a handful of b/w images; could this fit into your 29KB?

I reckon I'll dig the HTML files out of my project directory and see how easy it is to convert to MuCho's parser format.

Sol_HSA · October 2016

MykeP wrote: »

Anyway my immediate thought when I saw this was; given the book was 118 pages (maybe 100 words on the longest page) with only a handful of b/w images; could this fit into your 29KB?

The compression ratios I've seen are around 50% for text (because there's not too much text to work with on a single page). So the 46k novella fit into the 29k, for example. If the raw text goes well beyond that, I doubt it'll fit. As for images, one can always just make them smaller and smaller... ;)

Ralf · October 2016

You need to do some good quality, full game to show what the engine is capable of and encourage people to use it.

And do it properly, with handdrawn, original graphics and not these ugly autoconverts.

Sol_HSA · October 2016

Ralf wrote: »

And do it properly, with handdrawn, original graphics and not these ugly autoconverts.

Are you volunteering? ;)

hikoki · October 2016

I like this mucho!!
Would these short gamebooks fit into memory?http://www.arborell.com/gamebook_archive.html

Sol_HSA · October 2016

Oh rats, I think I figured a way to make the compression better, even without changing engine code. I'll have to do some experiments and see.

Sol_HSA · October 2016

Yup, thanks to Einar again, got the compression improved, now my "Dukes" example has over 4k free space again. The text compression is closer to 50-60% of the original instead of 60-70% it used to be.

I won't push a new version of the kit just yet. I think I'll have to build something with it to find what kinds of features / bug fixes come up.

MykeP · October 2016

Knowing I'd be away in London this week, stuck in a hotel with time on my hands in the evenings, I figured I'd have a stab at this.

In 2 nights I've manually converted my HTML version of "The Deadly Shadow," stripping out the HTML tags from the description text as I went and converting the "Turn to page X..." links into MuCho's A$ pagenum format.

For now I've left the $I image.scr tags commented out but, amazingly, I don't seem to have made any mistakes and it compiled first time - huzzah! :D

Unfortunately the final line of output from mc.exe indicates bad news:

Total output size too large (29952 bytes max, 57724 found)

Even with your improved compression I think it'll struggle to get that below 29K and there are at least 26 images to add yet - I don't suppose you're considering a 128K version? ;)

Still, you've peaked my interest and I'm inspired to press on with this project.

Step 1. Rewrite Richard Brightfield's spy story in a simplified style.

Sol_HSA · October 2016

MykeP wrote: »

Total output size too large (29952 bytes max, 57724 found)

Whoopsie yeah. The total size is probably over 100k? Most of the gamebooks I've looked at are hopelessly too large.

And no, I'm not considering a 128k version, that would be a completely different can of worms.

Sol_HSA · October 2016

For those curious, the improved compression was achieved by asking zx7 to consider a block of data before the decompresison buffer for data to copy, and then filling that block with common strings from all of the pages.

To figure out what to put there, I first counted all instances of all the words from all the pages to get the top-N words, and then chained these words based on the most likely words that follow each word (within that top-N word group).

The result looks freaky:

the Back to be a couple of them were in which I had been clear that it was like 
an and then if you say, toward me, looking at me. He looked about me as they
have made my way down with it, for one who but he just what this cat's got out

I'm pretty sure there's an algorithm that would find the "optimal" dictionary, but I doubt it would perform dramatically better than what this one already does.

I do have to store this block of data, which makes the available space smaller, but it improves compression more than the space it takes, so it's a general win. In the "dukes" example the savings were over 3kB over not using the prefix block.

MykeP · October 2016

Sol_HSA wrote: »

Whoopsie yeah. The total size is probably over 100k?

Actually it's only 87,855 bytes and, if you could compress based on inclusion of the phrase "attaché case," I'd expect you could get it down to about 15K. ;)

Sol_HSA · October 2016

MykeP wrote: »

Sol_HSA wrote: »

Whoopsie yeah. The total size is probably over 100k?

Actually it's only 87,855 bytes and, if you could compress based on inclusion of the phrase "attaché case," I'd expect you could get it down to about 15K. ;)

It should already do that, thanks to zx7 =)

I'm currently fighting to get a 61672 byte storybook fit, and I'm currently off by 2183 bytes. I'm afraid I may have to chop off a few pages to get it to fit. And even now I need to make some rather big changes to the engine to get that to run.

So, ~88k is probably not going to fit into 29k unless it's REALLY repetitive. =)

Another change I made (which requires the changes to the engine) is that I no longer compress pages by themselves, but keep adding pages until I hit the 4k limit and then compress them together. If similar pages are stored in sequence, they should compress pretty well.

I actually made an experiment where I used $O blocks to literally remove text, but the result was worse than when zx7 handled the deduplication, so that's pretty pointless.

And since it seems I'm diving back to the engine code, I'll probably remove the audio engine, add ability to run custom code so I can add the audio back that way, and might as well add integer counter handling and make the whole thing turing complete while at it..

..don't expect a new version soon.

Sol_HSA · October 2016

Figured out a new way to compress a bit better.

In addition to the "let's pack N pages together" change mentioned above, I now check what pages compress well together. To do this, I compress all pages in pairs to see what yields the best compression ratio, and although I haven't yet tried to see if the result of the compilation works, the results are encouraging: earlier I had 2183 bytes too much, now I have 341 bytes free! And I didn't even need to remove any pages!

I figured that by putting the pages that have similar text close to each other in source, I should get better compression. Then I thought, why not automate this? And the best way to know how well different pages play with each other is to run them through the compressor and see the result. So the compression scheme now is:

while pages remain
  while buffer has space
    for all remaining pages
      does the page compress best with previously stored one?   
    store page to buffer
    mark page as used
  compress buffer

Again, there's probably some permutation which might compress best (like do a+b+c compress better than a+c+b?), but the results even with this scheme are pretty good.

start p80 p25 p1 zx7: 2822 -> 1354 (47.980%)
p33 p12 zx7: 2701 -> 1314 (48.649%)
p76 p13 zx7: 4084 -> 2174 (53.232%)
p71 p85 p36 p74 zx7: 3592 -> 1667 (46.409%)
p94 p19 p87 p81 zx7: 3615 -> 1572 (43.485%)
p3 p95 p35 p53 p21 zx7: 3962 -> 1803 (45.507%)
p77 p97 p23 p7 zx7: 3521 -> 1633 (46.379%)
p39 p49 p42 p72 p69 p10 zx7: 4070 -> 2091 (51.376%)
p4 p70 p50 p48 p62 p24 zx7: 3943 -> 1510 (38.296%)
p52 p18 p60 p78 p68 p43 p58 zx7: 3892 -> 1739 (44.681%)
p38 p92 p47 p96 p41 zx7: 4040 -> 1596 (39.505%)
p2 p63 p83 p8 p61 p30 p82 zx7: 3716 -> 1692 (45.533%)
p90 p51 p16 p15 p45 p46 zx7: 3839 -> 2061 (53.686%)
p73 p65 p99 p14 p88 p56 p93 p37 p98 p9 p22 zx7: 3981 -> 1960 (49.234%)
p89 p20 p11 p75 p91 p27 p44 p66 p28 p100 p6 p86 p84 zx7: 4076 -> 2206 (54.122%)
p29 p67 p55 p34 p54 p32 p17 p57 p5 p64 p31 p40 p59 p26 p79 zx7: 3962 -> 2011 (50.757%)

Note the sets that go down under 40%! And to think I started off being relatively happy with 60-70%.. =)

Actually, now that I think about it, I might get even better results by not compressing with the previous page but with the current output buffer contents.. Hmm..

Sol_HSA · October 2016

..well, that worked, but not as well as the find-pairs one.

start p34 p59 p55 p31 p56 p89 p22 p67 p29 p40 p98 p26 p79 p20 p6 p9 p11 zx7: 3902 -> 1815 (46.515%), 0x5bcc
p100 p64 p63 p2 p14 p99 p37 p84 p44 p54 p32 p86 zx7: 3807 -> 1881 (49.409%), 0x62e3
p5 p57 p17 p46 p69 p10 p4 p70 p68 zx7: 3882 -> 1509 (38.872%), 0x6a3c
p91 p65 p73 p75 p27 p28 p61 p30 p66 p15 zx7: 4072 -> 2190 (53.782%), 0x7021
p78 p52 p24 p60 p72 p18 p96 p93 zx7: 4094 -> 1781 (43.503%), 0x78af
p36 p74 p85 p39 p3 p95 zx7: 3551 -> 1459 (41.087%), 0x7fa4
p88 p16 p41 p83 p8 p81 zx7: 3968 -> 1913 (48.211%), 0x8557
p82 p97 p62 p48 p51 zx7: 3437 -> 1654 (48.123%), 0x8cd0
p92 p38 p58 p7 zx7: 3406 -> 1405 (41.251%), 0x9346
p45 p53 p21 p77 p90 zx7: 4019 -> 1928 (47.972%), 0x98c3
p47 p87 p50 p42 zx7: 3417 -> 1749 (51.185%), 0xa04b
p49 p43 p19 p94 zx7: 3951 -> 1917 (48.519%), 0xa720
p71 p33 p23 zx7: 3246 -> 1685 (51.910%), 0xae9d
p35 p80 p25 zx7: 4039 -> 1987 (49.195%), 0xb532
p76 p13 zx7: 2730 -> 1412 (51.722%), 0xbcf5
p1 zx7: 1611 ->  785 (48.727%), 0xc279
p12 zx7: 2756 -> 1468 (53.266%), 0xc58a

Probably because this didn't look for best compression ratio, but rather absolute smallest addition, so it packs the smallest pages first, rather than seeing how different pages compress together.

Sol_HSA · October 2016

Okay, this surprised me.

When starting to compress a new buffer, which page should we choose as the first? My guess was "biggest", so I tried that. Made things worse. If I started from the biggest page, but let the "reset" be random, I got slightly better compression (314 bytes free, up from 270 when we start from page 0).

Then I figured, what if I start from the page that has best compression ratio when combines with another page?

1620 bytes free. I was floored.

p21 p80 p25 p1 zx7: 3666 -> 1776 (48.445%), 0x5bcc
p33 p12 zx7: 1973 -> 1044 (52.914%), 0x62bc
p76 p13 p71 zx7: 3613 -> 1889 (52.283%), 0x66d0
p85 p36 p74 p94 p19 zx7: 3762 -> 1722 (45.774%), 0x6e31
p87 p81 p3 p95 p35 zx7: 3718 -> 1694 (45.562%), 0x74eb
p53 p23 p77 p97 zx7: 3655 -> 1925 (52.668%), 0x7b89
p51 p42 p72 p69 p10 p4 zx7: 4029 -> 1919 (47.630%), 0x830e
p70 p50 p48 p62 p24 zx7: 3782 -> 1674 (44.262%), 0x8a8d
p52 p18 p60 p78 p68 p39 p49 zx7: 3960 -> 1848 (46.667%), 0x9117
p61 p30 p92 p38 p58 zx7: 3409 -> 1403 (41.156%), 0x984f
p43 p47 p96 p41 p2 p63 zx7: 4038 -> 1881 (46.582%), 0x9dca
p83 p8 p90 p82 p7 zx7: 3602 -> 1403 (38.951%), 0xa523
p16 p15 p45 p46 p73 p65 p99 p14 p88 zx7: 3805 -> 1964 (51.616%), 0xaa9e
p56 p93 p37 p98 p9 p22 p89 p20 p11 p75 p91 p27 p44 zx7: 3860 -> 1976 (51.192%), 0xb24a
p66 p28 p100 p6 p86 p84 p29 p67 p55 p34 p54 p32 p17 p57 zx7: 4048 -> 2116 (52.273%), 0xba02
p5 p64 p31 p40 p59 p26 p79 start zx7: 1804 ->  870 (48.226%), 0xc246

It DOES mean we're compressing everything with everything, which is tad bit on the slow side, especially since I don't currently cache the results. But that was rather effective.

Sol_HSA · October 2016

Okay, last update for the day. Promise. I put in the caching of the compression results which speeds things up as expected, but I also noticed that I had a few bugs thanks to bad variable naming.

p21 p77 p80 p25 zx7: 3261 -> 1414 (43.361%), 0x5bcc
p58 p38 p92 p13 zx7: 3591 -> 1032 (28.739%), 0x6152
p74 p36 p85 p71 p1 zx7: 3319 -> 1139 (34.318%), 0x655a
p94 p19 p12 zx7: 3173 -> 1231 (38.796%), 0x69cd
p10 p69 p70 p4 p50 p48 zx7: 3466 -> 1082 (31.218%), 0x6e9c
p83 p8 p76 p33 zx7: 3504 -> 1321 (37.700%), 0x72d6
p96 p47 p2 p41 p23 zx7: 3203 ->  984 (30.721%), 0x77ff
p99 p14 p35 p53 p42 p72 zx7: 3807 -> 1713 (44.996%), 0x7bd7
p95 p3 p87 p81 p61 p30 zx7: 3885 -> 1414 (36.396%), 0x8288
p52 p18 p24 p62 p60 p78 p68 zx7: 3898 -> 1375 (35.274%), 0x880e
p49 p39 p82 p43 p97 zx7: 3599 -> 1634 (45.402%), 0x8d6d
p79 p26 p51 p90 p45 p7 p16 p15 zx7: 4095 -> 2115 (51.648%), 0x93cf
p89 p22 p6 p9 p98 p63 p88 p56 p93 p73 p46 p65 zx7: 3948 -> 1823 (46.175%), 0x9c12
p75 p11 p91 p27 p20 p100 p66 p28 p54 p32 p37 p29 zx7: 3957 -> 2112 (53.374%), 0xa331
p55 p34 p86 p84 p31 p40 p67 p59 p64 p5 p57 p17 p44 start zx7: 3458 -> 1777 (51.388%), 0xab71

6558 bytes free.

That's so ridiculously huge value, I'm not sure if things work correctly or if I'm dropping data on the floor or something. Only way to know is to update the engine to see if it can handle the data. Anyway, look at those compression ratios - below 30% in some cases! That's nuts.

Sol_HSA · October 2016

Yup, that was too good to be true. Due to mixup in indices some pages were used several times and others not used at all. Good news is, the story I'm trying to fit still seems to fit, even though the free space is only 369 bytes.

p21 p77 p80 zx7: 3261 -> 1414 (43.361%), 0x5bce
p38 p58 p92 zx7: 2708 -> 1028 (37.962%), 0x6154
p36 p74 p85 p71 zx7: 2774 -> 1118 (40.303%), 0x6558
p19 p94 p1 zx7: 3791 -> 1735 (45.766%), 0x69b6
p69 p10 p4 p70 p50 zx7: 2994 -> 1066 (35.605%), 0x707d
p48 p62 zx7: 1421 ->  598 (42.083%), 0x74a7
p8 p83 zx7: 1415 ->  605 (42.756%), 0x76fd
p47 p96 p12 zx7: 4088 -> 2029 (49.633%), 0x795a
p14 p99 p25 p76 zx7: 3350 -> 1649 (49.224%), 0x8147
p3 p95 p81 p13 zx7: 3386 -> 1579 (46.633%), 0x87b8
p35 p53 p33 p42 zx7: 3950 -> 1974 (49.975%), 0x8de3
p18 p52 p24 p72 p60 p78 p68 zx7: 3546 -> 1455 (41.032%), 0x9599
p7 p23 p61 p30 p49 zx7: 3539 -> 1825 (51.568%), 0x9b48
p26 p79 p39 p82 p43 p87 zx7: 3444 -> 1812 (52.613%), 0xa269
p2 p41 p63 p51 p97 p90 zx7: 3542 -> 1800 (50.819%), 0xa97d
p22 p89 p20 p45 p46 p73 p65 p66 p16 p15 zx7: 3995 -> 2105 (52.691%), 0xb085
p9 p98 p37 p29 p67 p88 p56 p93 p64 p75 p11 p91 zx7: 3835 -> 1936 (50.482%), 0xb8be
p34 p55 p31 p84 p6 p86 p100 p28 p54 p32 p17 p57 p5 zx7: 3772 -> 1971 (52.253%), 0xc04e
p27 p44 p40 p59 start zx7: 1077 ->  654 (60.724%), 0xc801

Sigh.

Sol_HSA · October 2016

Implemented custom code blocks:

$C beepfx.ihx 27 !someflag

If someflag is not on, unpack beepfx code to 0xd000 and call it with HL=27.

I had to ditch the beepfx code from the engine proper in order to make room for new features.

Speaking of new features, integer ops also implemented, meaning you can do stuff like x+1, x==3, x-=y, x<=y, etc, used in many gamebooks. I have to figure out some sane way to print out the values of those variables though.

Still a lot to do (and documentation on top of that) before the next kit.. and I still haven't been able to reach the gamebook author. :(

Sol_HSA · October 2016

New features mostly done and largely working, I know of at least one bug I have still to figure out, and then there's also documentation.. Large amount of time went to refactoring the compiler, and implementing go and goto instructions. Which will be really fun to document... =)

Anyway, a teaser:

Sol_HSA · October 2016

WIP:

online emu link (works in firefox)
http://torinak.com/qaop#l=https://dl.dropboxusercontent.com/u/75238938/waiting.tap

tap file
https://dl.dropboxusercontent.com/u/75238938/waiting.tap

Sol_HSA · November 2016

Got permission from the "Waiting for the Light" author to use his gamebook, so I can finally go forward with the release of the third (and hopefully final) version of the MuCho development kit. Still have a couple examples to tweak.. possibly tomorrow?

MuCho multiple choise adventure engine, release 2 - little to no programming needed

Comments