New tool: ZXBOOT.EXE

ASMASM
edited January 2015 in Announcements
ASM is proud to present the first release of ZXBOOT.EXE: http://www.4shared.com/file/NP-FVOSf/zxboot__beta_.html

ZXBOOT is a command line tool for PCs running the Windows operating system. Its role is to compress 48K .SNA snapshot files so they can be loaded or saved using Spectrum BASIC. A self extract method is used to depack and run the game/program once the RANDOMIZE USR command is used. The output of ZXBOOT is 44100hz 8 BIT unsigned mono WAV or binary.

The output files may be loaded into a real Spectrum or using an emulator. The org address is the same as the execute address. The PROGRAM load address can be calculated by simply subtracting the file size from 65536. BASIC will load the .WAV files at the correct address but the .BIN files will need the actual org address. The following files are saved in the current directory:

SCREEN.WAV
PROGRAM.WAV

and/or

SCREEN.BIN
PROGRAM.BIN

A BASIC listing can be displayed by the application showing both the correct CLEAR and RANDOMIZE USR values for that specific compressed snapshot. The program file should be loaded first and the screen file last as BASIC will corrupt the display by printing the filename during load.

ZXBOOT uses the stack and repeated areas of memory to hide compact Z80 code including the register setup, depack and clean up routines. The program data is compressed so that it may be loaded into memory with plenty of space for the system variables and BASIC program area. The screen is left intact with only 17 BYTES of Z80 code left on the top of the stack once the game/program executes.

Please report bugs on this thread.

Regards
Post edited by ASM on
«13

Comments

  • edited November 2010
    Wow, thank you asm, this looks amazingly good and RLE'ish compression works very nicely.

    I guess I cannot beat you with a crappy gui :) I'm trying to inject bitbuster into my program, but it looks like rle also works nice. I also realized we can do this in one code block (by stitching compressed screen file to empty area) with compression (on some games).


    But here is a sna, that won't work with your program:
    http://arda.kisafilm.org/fish/batman1.sna

    it does not compress at all, so no space for basic to run.
  • edited November 2010
    Maybe I'm a bit dumb but...

    .SNA file contains whole RAM memory of Spectrum - 49152 bytes plus a few bytes for stored registers and other header info.

    When we compress it, it will be smaller.

    But when we decompress it in Spectrum memory it will decompress into the whole memory and eventually overwrite the running decompressing routine which means crash.

    Where am I wrong?
  • edited November 2010
    Ralf wrote: »
    But when we decompress it in Spectrum memory it will decompress into the whole memory and eventually overwrite the running decompressing routine which means crash.

    Where am I wrong?

    we can keep the decompresser in screen$ area or just below stack so we stay safe. But in practice you are right, it's impossible to do it without corrupting memory a little bit.
  • edited November 2010
    Nice job ASM,
    Ralf wrote: »
    Maybe I'm a bit dumb but...

    .SNA file contains whole RAM memory of Spectrum - 49152 bytes plus a few bytes for stored registers and other header info.

    When we compress it, it will be smaller.

    But when we decompress it in Spectrum memory it will decompress into the whole memory and eventually overwrite the running decompressing routine which means crash.

    Where am I wrong?

    Ralf, your not being dumb, everything you have said is pretty much true. What asm has done is very clever in the order that stuff is uncompressed so that the last bit of extraction leaves 17 bytes on the stack that wouldn't have otherwise been there. That 17 bytes will be decoding the last block right at the top of memory. 17 bytes is a mall price to pay for a clever bit of coding which 99.5% of programs wouldn't be affected by.
  • ASMASM
    edited November 2010
    Thanks for the comments.

    Here is a quick explanation of the compression system:

    As many of you know, 48K .SNA files have a 27 BYTE header followed by an uncompressed snapshot of the upper 49152 BYTES of the Spectrum memory map. The screen occupies the first 6912 followed by the system variables at 23296. As I am not interested in the screen, the area processed is from 23296 to 65535 (42240 BYTES). The screen is left alone as it is easier to just load it into the normal location of 16384 and forget about it.

    The first thing to do is to check the SP and make sure that 17 BYTES can be placed on the stack top without entering the ROM area. Once the stack is verified it is filled with a specific pattern of 17 BYTES to make sure that it will be stored as a copy because if any of those 17 BYTES become stored as a repeated region then 'part3' of the boot process could potentially wipe itself causing a crash - trust me on this! The stack area can be anywere within those 42240 BYTES as long as it is stored as a copy. 'Part3' sits in the stack and is called last to restore the repeated memory region used by 'part2'. 'Part1' is overwritten immediately after the execution of 'part2'...

    The next job is to profile the 42240 BYTES looking for repeated regions. If a region can be found that is equal or larger than 47 BYTES then we have somewhere to store 'part2' of the boot code. ZXBOOT looks for the very last section of 47 or more BYTES so that 'part2' can be placed high in memory. 'Part2' is the depack routine. It can never decompress over itself so it has to be as high as possible in memory. The clever part of the compression is the way that 'copy' and 'fill' sections alternate and the last ever depack is a copy. Here is an example; if the last 256 BYTES of memory were set to 255 then the 47 BYTES of 'part2' would reduce those 256 BYTES to 209 BYTES. Those missing 47 BYTES would be after the 209 BYTES thus placing them right at the end of memory.

    The very last depack never happens as the data is already in the correct place and there is no way to store the final counter for those BYTES. If the depacker continued then it would overwrite itself. To detect this and stop the depacker we must check for the source and destination pointers being the same value! The previous depack cycle would have been a repeated fill so that area of memory would have been reduced to 3 BYTES in packed form and fill the rest of the memory up to the last section when uncompressed. There is a problem with this method as final compression size is decreased depending on how low 'part2' resides in memory.

    I have improved the method to compress repeated regions less than 47 BYTES as anything lower was ignored and stored as a 'copy'. I have tried '5' as the threshold and that saves 1294 BYTES on BATMAN1.SNA. I will look into better compression but those 47 BYTES for 'part2' will increase in size depending on the method I choose...

    Here is the Z80 code for 'part1', 'part2' and 'part3':
    ; --------------------------------------------------------------------------
    
    
    ; PART1: Z80 code to initialize snapshot and execute PART2 (47 BYTES).
    
    
    z80_boot	db	243		; di
    stack		db	49,0,0		; ld	sp,0
    
    border		db	62,0		; ld 	a,0
    		db	211,254		; out	(254),a
    
    reg_r		db	62,0		; ld	a,0
    		db	0EDh,79		; ld	r,a
    	
    reg_i		db	62,0		; ld	a,0
    		db	0EDh,71		; ld	i,a
    	
    intmode		db	0EDh,94		; im2
    
    reg_ix		db	0DDh,33,0,0	; ld	ix,0
    reg_iy		db	0FDh,33,0,0	; ld	iy,0
    
    reg_hlx		db	33,0,0		; ld	hl,0
    reg_dex		db	17,0,0		; ld	de,0
    reg_bcx		db	1,0,0		; ld	bc,0
    		db	217		; exx
    
    		db	241		; pop	af
    		db	8		; ex	ex,af'
    
    source		db	33,0,0		; ld	hl,0
    		db	17,0,91		; ld	de,23296
    execute		db	195,0,0		; jp	0
    
    
    ;---------------------------------------------------------------------------
    
    
    ; PART2: Z80 code to depack snapshot and execute PART3 (47 BYTES).
    
    
    z80_stub	db	78		; ld	c,(hl)
    		db	35		; inc	hl
    		db	70		; ld	b,(hl)
    		db	35		; inc	hl
    
    		db	203,120		; bit	7,b
    		db	40,00Ch		; jr	z,.copy
    
    ;		------------
    
    		db	203,184		; res	7,b
    
    		db	237,160		; ldi
    		db	43		; dec	hl
    
    		db	121		; ld	a,c
    		db	176		; or	a,b
    		db	32,0F9h		; jr	nz,.fill
    	
    		db	35		; inc	hl
    		db	24,002h		; jr	.next
    
    ;		------------
    
    		db	237,176		; ldir
    
    		db	123		; ld	a,e
    		db	189		; cp	a,l
    		db	32,0E6h		; jr	nz,.region
    
    		db	122		; ld	a,d
    		db	188		; cp	a,h
    		db	32,0E2h		; jr	nz,.region
    	
    ;		------------
    
    		db	241		; pop	af
    
    restore		db	33,0,0		; ld	hl,0
    reg_de		db	17,0,0		; ld	de,0
    value		db	1,253,STUBLEN	; ld	bc,0
    
    jump		db	195,0,0		; jp	0
    	
    reg_stack	db	0,0,0,0		; Registers (4 BYTES)
    
    
    ; --------------------------------------------------------------------------
    
    
    ; PART3: Z80 code to clean up memory used by PART2 (17 BYTES).
    
    
    z80_stack	db	113		; ld	(hl),c
    		db	35		; inc	hl
    		db	16,0FCh		; djnz	.fill
    
    reg_hl		db	33,0,0		; ld	hl,0
    reg_bc		db	1,0,0		; ld	bc,0
    reg_sp		db	49,0,0		; ld	sp,0
    
    iff2		db	243		; di
    reg_pc		db	0,0,0		; retn / jp nn
    
    
    ; --------------------------------------------------------------------------
    

    Expect an update soon...
  • edited November 2010
    Fantastic!!!... Works a treat... Managed to Convert MoreTeaVicar.sna to TAP easily... Im curious as to what your update will consist of, as it already does a good job... How about /t for .tap output?... (for emulators - yeah, I know you hate them!!!!!)... ;)

    A job well done!!!!
  • edited November 2010
    So basically the algorithm it uses to "inject" itself into stuff is not unlike that of the CIH virus, is it? :)
    Also, hope you don't mind a tiny improvement for PART2:
    Before
    	ldi
    	dec hl
    	ld a,c
    	or b
    	jr nz,.fill
    
    After
    	ldi
    	dec hl
    	jp pe,.fill
    
  • ASMASM
    edited November 2010
    Hikaru wrote: »
    So basically the algorithm it uses to "inject" itself into stuff is not unlike that of the CIH virus, is it? :)
    Also, hope you don't mind a tiny improvement for PART2:
    Before
    	ldi
    	dec hl
    	ld a,c
    	or b
    	jr nz,.fill
    
    After
    	ldi
    	dec hl
    	jp pe,.fill
    

    Cool! I have learn't something new about the Z80 today as I never knew about the use of the overflow flag on LDI. It will save one BYTE but mean I will have to relocate the JP address in the x86 code...

    I have only started using LDI and LDIR now that I have matured as a programmer as during my teenage years I considered block copy instructions to be 'too slow' :D

    Cheers
    kgmcneil wrote: »
    Fantastic!!!... Works a treat... Managed to Convert MoreTeaVicar.sna to TAP easily... Im curious as to what your update will consist of, as it already does a good job... How about /t for .tap output?... (for emulators - yeah, I know you hate them!!!!!)... ;)

    A job well done!!!!

    The update will consist of better compression with RLE profiling to make sure that BYTES are not wasted. The use of 'TAP' for emulators would be a bit pointless as the data came from a standard emulator file (.SNA) in the first place. The use of WAV is for sending the data to a real Spectrum using standard media players - PlayTZX runs into problems on my system. A WAV file can be played by many desktop applications.
  • ASMASM
    edited November 2010
    Here is ZXBOOT.EXE version v1.0: http://www.4shared.com/file/Yz39ttJU/zxboot__v10_.html

    I decided to improved the compression system for ZXBOOT as Jon Ritman's Batman did not pack well. I initially used the profile data to pack the snapshot as it was convenient at the time. It turned out to be inefficient so I wrote a brand new method to pack the data.

    I have updated the RLE algorithm replacing the WORD count with a BYTE count and added a special 256 BYTE copy when the code 128 is encounted. I have also used code 0 for a terminator as that saved Z80 code space. The opcode 0 (NOP) must be added one BYTE before the depack code so the routine can stop before overwriting itself. As there is no room for the last RLE code, the terminator as to be part of the depack routine.

    The threshold for the RLE FILL has been seperated from the region profiler and set to 3 as that is optimum. Here are the codes for the new RLE:
           0 = TERMINATE
      1..127 = STREAM COPY
         128 = SPECIAL 256 BYTE STREAM COPY
    129..255 = REGION FILL
    

    This update took ten solid hours to develop and worked first time when I tested the output on Spectaculator. Obviously the x86 was tested in stages but as the Z80 worked first time I have to eat my socks :D

    Batman now packs to 40549 BYTES saving 1691 BYTES:
    File: BATMAN1.SNA 
    
    SP: 65520
    PC: 26177
    
    Threshold: 43 BYTES
    
    31757 - 31904 -   0 - 148
    41174 - 41216 -   0 - 43
    64432 - 64487 -   0 - 56
    64915 - 65066 -   0 - 152
    
    4 regions found
    
    1691 BYTES saved
    
    Length: 40549
    Origin: 24987
    
    10 CLEAR 24986
    20 LOAD "" CODE
    30 LOAD "" CODE
    40 RANDOMIZE USR 24987
    

    BTW - Snapshot Batman on a blank screen to save more space!


    Here is the x86 sourcecode for the new Z80 routines:
    ; --------------------------------------------------------------------------
    
    
    ; PART1: Z80 code to initialize and execute depack (48 BYTES).
    
    
    z80_boot	db	243		; di
    stack		db	49,0,0		; ld	sp,0
    
    border		db	62,0		; ld 	a,0
    		db	211,254		; out	(254),a
    
    reg_r		db	62,0		; ld	a,0
    		db	0EDh,79		; ld	r,a
    	
    reg_i		db	62,0		; ld	a,0
    		db	0EDh,71		; ld	i,a
    	
    intmode		db	0EDh,94		; im2
    
    reg_ix		db	0DDh,33,0,0	; ld	ix,0
    reg_iy		db	0FDh,33,0,0	; ld	iy,0
    
    reg_hlx		db	33,0,0		; ld	hl,0
    reg_dex		db	17,0,0		; ld	de,0
    reg_bcx		db	1,0,0		; ld	bc,0
    		db	217		; exx
    
    		db	241		; pop	af
    		db	8		; ex	ex,af'
    
    source		db	33,0,0		; ld	hl,0
    		db	17,0,91		; ld	de,23296
    		db	67		; ld	b,e
    execute		db	195,0,0		; jp	0
    
    
    ;---------------------------------------------------------------------------
    
    
    ; PART2: Z80 code to depack snapshot and execute clean up (43 BYTES).
    
    
    z80_stub	db	0		; nop
    
    		db	230,127		; and	a,127
    		db	71		; ld	b,a
    		db	40,8		; jr	z,.256
    
    		db	126		; ld	a,(hl)
    		db	35		; inc	hl
    
    		db	18		; ld	(de),a
    		db	19		; inc	de
    		db	16,0FCh		; djnz	.fill
    		db	24,4		; jr	.start
    
    		db	4		; inc	b
    
    		db	79		; ld	c,a
    		db	237,176		; ldir
    
    		db	126		; ld	a,(hl)
    		db	35		; inc	hl
    		db	167		; and	a,a
    signed		db	250,0,0		; jp	m,.signed
    		db	32,0F5h		; jr	nz,.copy
    
    		db	241		; pop	af
    
    restore		db	33,0,0		; ld	hl,0
    reg_de		db	17,0,0		; ld	de,0
    value		db	1,253,STUBLEN	; ld	bc,0
    
    jump		db	195,0,0		; jp	0
    	
    reg_stack	db	0,0,0,0		; Registers (4 BYTES)
    
    
    ; --------------------------------------------------------------------------
    
    
    ; PART3: Z80 code to restore repeated memory (17 BYTES).
    
    
    z80_stack	db	113		; ld	(hl),c
    		db	35		; inc	hl
    		db	16,0FCh		; djnz	.fill
    
    reg_hl		db	33,0,0		; ld	hl,0
    reg_bc		db	1,0,0		; ld	bc,0
    reg_sp		db	49,0,0		; ld	sp,0
    
    iff2		db	243		; di
    reg_pc		db	0,0,0		; retn / jp nn
    
    
    ;---------------------------------------------------------------------------
    

    Here is the new Z80 depack code with comments:
    ; --------------------------------------------------------------------------
    
    
    ; Z80 RLE depack routine.
    
    
    		nop			; This is the terminator
    
    .signed:	and	a,127		; Removed sign bit
    		ld	b,a		; Set for loop
    		jr	z,.256		; 0 = special 256 BYTE COPY
    
    		ld	a,(hl)		; Read FILL value
    		inc	hl
    
    .fill:		ld	(de),a		; FILL repeated region
    		inc	de
    		djnz	.fill
    		jr	.start		; Exit with B = 0
    
    ;		------------
    
    .256:		inc	b		; A = 0 so BC will equal 256
    
    .copy:		ld	c,a		; BC = count from stream
    		ldir			; COPY BC BYTES from stream
    
    .start:		ld	a,(hl)		; Entry is here, B must = 0
    		inc	hl
    		and	a,a		; Check sign of A?
    		jp	m,.signed	; Negative for repeated FILL
    		jr	nz,.copy	; 0 = end, 1 to 127 for COPY
    
    
    ; --------------------------------------------------------------------------
    

    As usual, report ZXBOOT bugs and suggestions on this thread.

    Regards

    ASM
  • jpjp
    edited November 2010
    This is very, very cool stuff - unbelievably useful!

    As ASM has suggested, the point at where the snapshot is taken is really important.
    I've been messing around with the Speedlock 7 release of Batman found here - by placing a breakpoint at $fe40, and making a snapshot the second time the breakpoint is hit, a few more bytes are shaved off:
    File: BATMAN.SNA
    
    SP: 26877
    PC: 65088
    
    Threshold: 43 BYTES
    
    31741 - 31904 - 204 - 164
    63548 - 63603 - 204 - 56
    64047 - 64255 -   0 - 209
    64302 - 64453 -   0 - 152
    
    4 regions found
    
    1880 BYTES saved
    
    Length: 40360
    Origin: 25176
    
    10 CLEAR 25175
    20 LOAD "" CODE
    30 LOAD "" CODE
    40 RANDOMIZE USR 25176
    

    Snapshot here
  • ASMASM
    edited November 2010
    jp wrote: »
    This is very, very cool stuff - unbelievably useful!

    As ASM has suggested, the point at where the snapshot is taken is really important.
    I've been messing around with the Speedlock 7 release of Batman found here - by placing a breakpoint at $fe40, and making a snapshot the second time the breakpoint is hit, a few more bytes are shaved off

    I'm glad this is being put to good use. You have beat my snapshot of Batman that I made just after the batmobile screen; 1864 BYTES.

    My version of Batman does not work on the +3 surprise surprise...

    It's a shame that I don't have time to do the 128K version of this system - maybe one day. I have to write the backwards version now as my RS232 transfer link occupies the last 512 BYTES of the Z80 memory map. In theory this system should be able to depack backwards from 65023 to 16384... at least I will have interrupts off and BASIC will be dead so I can depack the whole snapshot in one go :D
  • jpjp
    edited November 2010
    Hmm, there's something quite wrong with SNA itself - not sure what, but at the menu screen, keys are pressed for you and the game starts... :-o
  • ASMASM
    edited November 2010
    Here is ZXBOOT version v1.1: http://www.4shared.com/file/aTXPQQDK/zxboot__v11_.html

    I have added the switch /p to place a small 185 BYTE routine at the start of the EXE to restore the +3/2a ROM back to an original 48K Spectrum. The patch allows games such as Bomb Jack, Batman and Gryzor to work on a +3 or +2a an any mode. I have locked out the patch if 48K, 128K+ or +2 models are detected as it will crash. The patch also disables paging allowing some games to work in 128K mode - Sabre Wulf etc.

    RAM paging mode 4 5 6 3 has been used to change the contents of the ROM. The memory footprint of the game/program may increase in size as the patch routine must be placed below 32768 due to the memory remap algorithm using page 2 (32768..49151) for workspace. Games may run slower and time critical routines will fail as the first three 16K memory banks are contended.

    I have completely restored every BYTE of the original 48K ROM for full compatibility.
    ASM wrote: »
    I have to write the backwards version now as my RS232 transfer link occupies the last 512 BYTES of the Z80 memory map. In theory this system should be able to depack backwards from 65023 to 16384

    I found that I could place a simple 14 BYTE relocate routine before the EXE to allow it to work with my RS232 downloader - backwards RLE method... tut! tut! This ROM restore system took 10 hours work from start to finish, the Z80 was developed using PDS/2 and tested on a real +3. Spectaculator was used to check compatibility across all Spectrum models and also rip the ROMs for examination.


    Here is the 185 BYTE space optimized Z80 to restore the 48K ROM on a +3/2a machine.
    ; --------------------------------------------------------------------------
    
    
    ; Compact Z80 to restore +3/2a ROM back to an original 48K Spectrum.  
    
    
    start:		di
    		ld	sp,copy_tab		; **MUST RELOCATE**
    
    		ld	a,(14517)		; Spectrum +3/2a ROM?
    		and	a,a
    		jr	nz,skip
    
    		ld	bc,01FFDh
    		ld	a,00000100b		; Vertical ROM select/count
    		out	(c),a
    
    remap:		ld	bc,07FFDh		; Remap memory
    		pop	hl
    		dec	sp
    		pop	de
    		out	(c),l			; RAM page select
    
    		ld	bc,16384		; Memory page size
    		ld	l,c
    		ld	e,c
    		ldir
    
    		dec	a
    		jr	nz,remap
    
    ;		------------
    
    		ld	bc,01FFDh		; Page 4 5 6 3
    		ld	a,00000101b
    		out	(c),a
    		ld	b,07Fh			; Disable paging
    		ld	a,00100000b
    		out	(c),a
    
    ;		------------
    
    		ld	b,d			; Patch to 48K ROM
    		ex	de,hl			; HL = 0
    		add	hl,sp			; Point HL to change_tab
    
    		ld	a,11			; Number of sections
    
    patch:		ld	e,(hl)			; Read address into DE
    		inc	hl
    		ld	d,(hl)
    		inc	hl
    		ld	c,(hl)			; Read count into C, B = 0
    		inc	hl	
    		ldir
    		dec	a
    		jr	nz,patch
    
    		ld	hl,14446		; Fill last section with 255
    		ld	bc,1170-1
    		ldir
    
    skip:		jr	program
    
    
    ; --------------------------------------------------------------------------
    
    
    ; 12 BYTE memory remap table.
    
    
    copy_tab	db	00010100b,000,192	; Copy 48K ROM to page 4
    		db	00010110b,128,192	; Copy page 2 to page 6
    		db	00010000b,192,128	; Copy page 0 to page 2
    		db	00010011b,128,192	; Copy page 2 to page 3
    
    
    ; --------------------------------------------------------------------------
    
    
    ; 95 BYTE change table for Spectrum +3/2a ROM.
    
    
    change_tab	dw	00019
    		db	1
    
    		db	255
    
    		dw	00075
    		db	2
    
    		db	191
    		db	2
    
    		dw	00109
    		db	1
    
    		db	32
    
    		dw	02466
    		db	22
    	
    		db	83
    		db	116
    		db	97
    		db	114
    		db	116
    		db	32
    		db	116
    		db	97
    		db	112
    		db	101
    		db	44
    		db	32
    		db	116
    		db	104
    		db	101
    		db	110
    		db	32
    		db	112
    		db	114
    		db	101
    		db	115
    		db	115
    
    		dw	02898
    		db	4
    
    		db	214
    		db	165
    		db	48
    		db	9
    
    		dw	04937
    		db	4
    
    		db	175
    		db	17
    		db	54
    		db	21
    
    		dw	05440
    		db	21
    
    		db	83
    		db	105
    		db	110
    		db	99
    		db	108
    		db	97
    		db	105
    		db	114
    		db	32
    		db	82
    		db	101
    		db	115
    		db	101
    		db	97
    		db	114
    		db	99
    		db	104
    		db	32
    		db	76
    		db	116
    		db	228
    
    		dw	07037
    		db	4
    
    		db	253
    		db	203
    		db	10
    		db	126
    
    		dw	07156
    		db	3
    
    		db	223
    		db	254
    		db	13
    
    		dw	09798
    		db	3
    
    		db	205
    		db	142
    		db	2
    
    		dw	14446
    		db	1
    
    		db	255
    
    
    ; --------------------------------------------------------------------------
    
    
    program:
    
    
    ; --------------------------------------------------------------------------
    

    BTW - I may have to remove the check at location 14517 for hardware devices that shadow the ROM, any comments?

    Regards

    ASM
  • edited November 2010
    A feature request ...

    Given that it's a straightforward command line tool, how about making it multi platform for those of us who are not Microsoft customers? It doesn't sound like it should be tied to win32 specific things so hopefully it's just a matter of recompiling on the target OS.
  • edited November 2010
    This tool reminds me of a similar one made many years ago by Antonio Villena Godoy, aka Espineter (the author of Bacteria emulator). His tool compressed a .tap file into another with all the code compressed in the basic block. AFAIK it ignores the screen, analyzes the code that's loaded, and places itself in some free space. The compression used was a complex LZ routine, allowing great compression rates. It was called TAPC, and you can find it here with examples:

    http://www.speccy.org/metalbrain/tapc.zip

    and here's the source code:

    http://www.speccy.org/metalbrain/tapc_source.zip

    ...sorry Winston, it was DOS based. :razz:

    Antonio explained the compression and I analized the decompression routine in an article for MagazineZX #4, but it's all in spanish. You may see it here, and maybe automatic translation will make some sense out of it:
    http://www.speccy.org/wiki/programacion/ensamblador/compresor_tapc
  • ASMASM
    edited November 2010
    Winston wrote: »
    A feature request ...

    Given that it's a straightforward command line tool, how about making it multi platform for those of us who are not Microsoft customers? It doesn't sound like it should be tied to win32 specific things so hopefully it's just a matter of recompiling on the target OS.

    Um... I only develop for Microsoft operating systems and have never ever used or installed anything other than Windows or DOS during my 18 year period of owning PCs. I used Mac OSX for a while but I could never get the right button on the mouse the work :D

    The command line application is 100% x86 assembler with Microsoft specific API calls. Sorry dude :(
    Metalbrain wrote: »
    The compression used was a complex LZ routine, allowing great compression rates

    I don't use LZ compression as I need to make sure that the data can be depacked over itself. I also want to keep the depack Z80 as small as possible to maximize the chances of finding a free area, the current RLE depacker is about 25 BYTES.

    Designing a CODEC that is capable of overwritting itself is a black art. I already found a bug that was introduced into ZXBOOT v1.0 / v1.1 - there is a case where the depack will fail on the last section as the count can be overwritten. I have redesigned the compressor but as the real world is demanding my attention, I have to go and do 'non development' activities for a while...

    I could solve the issue with a bodge but I would rather fix it properly.

    Expect an update over the next couple of days...
  • ASMASM
    edited November 2010
    Here is ZXBOOT version v1.2: http://www.4shared.com/file/LfEPqb8n/zxboot__v12_.html

    I have fixed the RLE decompression issue, it wasn't easy but the result is good. I have added a warning message when the memory safety limit is exceeded. RLE can work within a couple of BYTES of itself but as a precaution, I have used a simple rule that uses the packet count as a workspace limit. For a better understanding... each packet requires one BYTE of memory to be added to the size of the original data so using the packet count as a gap between the input stream and output stream makes sure that the RLE can never overtake itself.

    Here are the changes made to v1.2:

    Changed threshold from 43 BYTES to 46 BYTES.

    Fixed an issue that caused the decompression to fail under certain conditions.

    Added warning message when compression safety margin is exceeded.

    Added new command /d to disable memory paging for compatibility issues on 128K machines.

    Added new command /f to force +3/2a 48K ROM patch.

    Here is the current command set:
     ZXBOOT v1.2 - Copyright (c) 1996-2010 ASM.
     Sinclair ZX Spectrum 48K snapshot boot tool.
     usage: zxboot [options] [filename]
       
      /i - Display information.
      /r - Display regions.
       
      /d - Disable 128K memory paging.
      /f - Forced +3/2a to patch 48K ROM.
      /p - Detect +3/2a to patch 48K ROM.
       
      /b - Output BIN.
      /w - Output WAV (default).
       
      /? - Help.
    

    Regards

    ASM
  • edited November 2010
    ASM wrote: »
    The command line application is 100% x86 assembler with Microsoft specific API calls. Sorry dude :(

    I have to wonder ... why use Win32-only API calls when I'm certain the standard C library does everything that you need (and can be called from assembler)? It'd be then just a matter of recompiling for other platforms so long as they use x86.
  • ASMASM
    edited November 2010
    Winston wrote: »
    I have to wonder ... why use Win32-only API calls when I'm certain the standard C library does everything that you need (and can be called from assembler)? It'd be then just a matter of recompiling for other platforms so long as they use x86.

    Um.. I'm not into 'C' librarys at all as I don't care for the language one bit - In fact I hate it with a passion! My Windows command line tools and desktop applications use a shell provided by a programming buddy :D He decides on the linker and library includes and chooses the Windows API calls and wraps them for assembler use; i.e. removing that stupid PUSH parameters on the stack nackers. I don't believe that I call one single 'C' library function.

    I really hate having to search through MSDN when I need something that is missing - I just get him to sort it! As long as I have a screen pointer and basic I/O then I'm sorted as I write everything else myself... It is not my department as far as I'm concerned.

    I had a word last night with my programming bud and he raised some issues that would make it a problem with using Linux... I can't remember what they were as my eyes start rolling when non Microsoft operating systems are mentioned :rolleyes:

    I'm against open source so I don't give sourcecode out. I don't mind bits and pieces to provide help and examples but not full projects.

    As for 'C', I can't even remember how to use PRINTF - that's how much I hate it'!!! Only allowing one parameter to be passed back from a routine - who thinks this s*** up???

    Ooops!

    Winston is now stomping around the room kicking things :D

    Seriously - if someone can supply a basic shell for I/O then I could probably port my code... arrrh that was it - there can be a problem with giving out .EXEs on Linux as they may need to be rebuilt under certain conditions.

    Here are the Windows function calls:
    	extern	ExitProcess                    
    	extern	GetCommandLineA                
    	extern	GetStdHandle                   
    	extern	WriteFile 
    	extern	GlobalAlloc
    	extern	GlobalFree
    	extern	mmioOpenA
    	extern	mmioClose
    	extern	mmioRead
    	extern	mmioWrite
    	extern	mmioSeek
    	extern	DeleteFileA
    	extern	GetCurrentDirectoryA
    	extern	SetCurrentDirectoryA
    	extern	CreateDirectoryA
    	extern	RemoveDirectoryA
    

    Here is the bat file to build the project:
    @echo off
    
    echo Building Project... > err
    
    nasmw -O1 asm\main.asm -o obj\main.obj -fwin32 -s >> err
    
    alink -c -oPE -subsys con -entry WSTART lib\win32.lib obj\main.obj -o c:\utils\zxboot.exe >> err
    
    type err
    

    Here is MAIN.ASM:
    ; --------------------------------------------------------------------------
    
    
    	%include	"inc\equates.inc"
    
    	%include	"asm\washell.asm"	; Shell
    	%include	"asm\shellwin.asm"
    	%include	"asm\program.asm"
    
    	%include	"asm\cmdtail.asm"
    
    
    ; --------------------------------------------------------------------------
    
    
    	%include	"inc\data.inc"		; Data
    	%include	"inc\incbins.inc"
    
    
    ; --------------------------------------------------------------------------
    
    
    	%include	"asm\zxwave.asm"	; WAV
    	%include	"asm\zxboot.asm"	; Program
    
    
    ; --------------------------------------------------------------------------
    

    I'm not that bad really - honest - there are some subjects that set me off on a rant (just like any other nerd) and those are C/C++, Java, Apple, Indie music, facebook, TV soaps, BING!/MSN, Sony, Linux and open source :D

    EDIT:

    RIGHT! As the WOS forum as become a battle ground recently it ends here!!! I have said what I don't like as an example, this thread is not going to be Microsoft vs the world... How about I compromise and create a nice LIB with the functions needed for ZXBOOT so other members can create their own applications, DOS, Linux or Windows etc?
  • edited November 2010
    What's wrong with Indian music?
    I wanna tell you a story 'bout a woman I know...
  • edited November 2010
    Now now, he said Indie - so called "Independant music", but I suspect you knew that and are pulling ASM's whatsit in a playfull manner :smile:
  • edited November 2010
    MrCheese wrote: »
    Now now, he said Indie - so called "Independant music", but I suspect you knew that and are pulling ASM's whatsit in a playfull manner :smile:
    'Playful' is my middle name...
    I wanna tell you a story 'bout a woman I know...
  • edited November 2010
    karingal wrote: »
    What's wrong with Indian music?

    That reminds me of one of my drunken stories, all 100% true

    Was on the way to our usual 80's club in Reading, got a taxi, sikh driver, some crappy radio was blasting out but he had lots of tapes in his taxi, said to the guy (we're all quite drunk) have you got any indie music ? (obvious looking back he wasnt going to have any, like he'll say 'yeah i've got some Echobelly here...)

    Then he puts in a tape and we think result. Then of course we hear some indian bangra type stuff as he obviously thought we meant indian. We were too polite to say 'nah thats not what we meant' so we sat in silence for the next 15 minutes trying not to laugh as we go to Reading listening to this indian chanting stuff
  • edited November 2010
    psj3809 wrote: »
    That reminds me of one of my drunken stories, all 100% true

    Was on the way to our usual 80's club in Reading, got a taxi, sikh driver, some crappy radio was blasting out but he had lots of tapes in his taxi, said to the guy (we're all quite drunk) have you got any indie music ? (obvious looking back he wasnt going to have any, like he'll say 'yeah i've got some Echobelly here...)

    Then he puts in a tape and we think result. Then of course we hear some indian bangra type stuff as he obviously thought we meant indian. We were too polite to say 'nah thats not what we meant' so we sat in silence for the next 15 minutes trying not to laugh as we go to Reading listening to this indian chanting stuff
    lol, good story.
    I wanna tell you a story 'bout a woman I know...
  • edited November 2010
    ASM wrote: »
    As for 'C', I can't even remember how to use PRINTF - that's how much I hate it'!!!
    Yeah, it'll probably never catch on anyway. :)
  • edited November 2010
    I can get it to run under Darwine on OS X. It would be nice to have a native version though. If you could provide main.obj in macho format (does your nasm support the '-f macho' flag?), and someone creates a win32.lib equivalent, would that be enough? It should be easy enough to wrap up POSIX/libc calls to do the equivalent of your Win32 calls.
  • ASMASM
    edited November 2010
    JamesW wrote: »
    I can get it to run under Darwine on OS X. It would be nice to have a native version though. If you could provide main.obj in macho format (does your nasm support the '-f macho' flag?), and someone creates a win32.lib equivalent, would that be enough? It should be easy enough to wrap up POSIX/libc calls to do the equivalent of your Win32 calls.

    I know next to nothing about non Microsoft operating systems or building code for them. I just write x86 code and build with nasm and link to LIBs to call out to extern function such as Windows API functions.

    When it comes to developing for the PC, I usually use building blocks from other programmers to deal with the host operating system. I know next to nothing about Windows API functions. I know more about MS-DOS than Win32. My programming side kick has supplied me with the shells and build environments to run under DOS and Windows since 1995... I'm more into creating applications/tools/games rather than worring about how to write to the registry for example.

    Now if you give me an hardware manual and an empty machine, I can quite happily write device drivers once the machine has been cold started. I have many many library routines of my own without having to rely on someone else's function calls... I did this with the Spectrum, Amiga, Mega-Drive/Mega-CD, PCX, N64, GameBoy Color and PC motherboards :D I find desktop operating systems a pain in the arse but then we need 'em...

    As I've got older, I've found that I'm more into creating than learning new systems... The typical computer industry attitude is to throw away last weeks hardware/software and start again but use the excuse that we can recycle most of the sourcecode and lamely translate it into this weeks CPU or virtual machine... NO THANKS!!! I got sick and tired of just getting good at a system and having to bin it because, well just because... that's what the boss wanted!!!! ARRRRRRRRRRRRRRRRH!!!

    I got really good at Amiga after I let go of the Spectrum days but nooooooooo, that was no good now.... so I had to learn Mega-Drive but noooooo, then it was MS-DOS but nooooooooooo, it had to be Windows 95.... then Direct-X.... then C++, now god knows what..... STICK IT!!!!!!!!!

    So how do we solve running my x86 code on a Mac, Linux or the mircowave in the kitchen??? Maybe I shall give out the algorithm and someone can multi-platform development can just build it to run on everything including granny Schumacher's pet tortoise's mobile phone :D

    Here is the nasm help:
    usage: nasm [-@ response file] [-o outfile] [-f format] [-l listfile]
                [options...] [--] filename
        or nasm -r   for version info (obsolete)
        or nasm -v   for version info (preferred)
    
        -t          Assemble in SciTech TASM compatible mode
        -g          Generate debug information in selected format.
        -e          preprocess only (writes output to stdout by default)
        -a          don't preprocess (assemble only)
        -M          generate Makefile dependencies on stdout
    
        -E<file>    redirect error messages to file
        -s          redirect error messages to stdout
    
        -F format   select a debugging format
    
        -I<path>    adds a pathname to the include file path
        -O<digit>   optimize branch offsets (-O0 disables, default)
        -P<file>    pre-includes a file
        -D<macro>[=<value>] pre-defines a macro
        -U<macro>   undefines a macro
        -X<format>  specifies error reporting format (gnu or vc)
        -w+foo      enables warnings about foo; -w-foo disables them
    where foo can be:
        macro-params            macro calls with wrong no. of params (default off)
        macro-selfref           cyclic macro self-references (default off)
        orphan-labels           labels alone on lines without trailing `:' (default off)
        number-overflow         numeric constants greater than 0xFFFFFFFF (default on)
        gnu-elf-extensions      using 8- or 16-bit relocation in ELF, a GNU extension (default off)
    
    response files should contain command line parameters, one per line.
    
    valid output formats for -f are (`*' denotes default):
      * bin       flat-form binary files (e.g. DOS .COM, .SYS)
        aout      Linux a.out object files
        aoutb     NetBSD/FreeBSD a.out object files
        coff      COFF (i386) object files (e.g. DJGPP for DOS)
        elf       ELF32 (i386) object files (e.g. Linux)
        as86      Linux as86 (bin86 version 0.3) object files
        obj       MS-DOS 16-bit/32-bit OMF object files
        win32     Microsoft Win32 (i386) object files
        rdf       Relocatable Dynamic Object File Format v2.0
        ieee      IEEE-695 (LADsoft variant) object file format
    
  • edited November 2010
    ASM wrote: »
    Here is the nasm help:

    So what does:
    nasm -O1 asm\main.asm -o obj\main.obj -f macho -s >> err

    do? If it emits a main.obj we might be in business. If not then you're nasm doesn't do OS X executable formats and it's game over.
  • ASMASM
    edited November 2010
    JamesW wrote: »
    So what does:


    do? If it emits a main.obj we might be in business. If not then you're nasm doesn't do OS X executable formats and it's game over.

    Sorry man, unrecognised output 'macho'.

    EDIT:

    I just downloaded the latest nasm and it does do macho32/macho64.
    usage: nasm [-@ response file] [-o outfile] [-f format] [-l listfile]
                [options...] [--] filename
        or nasm -v   for version info
    
        -t          assemble in SciTech TASM compatible mode
        -g          generate debug information in selected format
        -E (or -e)  preprocess only (writes output to stdout by default)
        -a          don't preprocess (assemble only)
        -M          generate Makefile dependencies on stdout
        -MG         d:o, missing files assumed generated
        -MF <file>  set Makefile dependency file
        -MD <file>  assemble and generate dependencies
        -MT <file>  dependency target name
        -MQ <file>  dependency target name (quoted)
        -MP         emit phony target
    
        -Z<file>    redirect error messages to file
        -s          redirect error messages to stdout
    
        -F format   select a debugging format
    
        -I<path>    adds a pathname to the include file path
        -O<digit>   optimize branch offsets
                    -O0: No optimization (default)
                    -O1: Minimal optimization
                    -Ox: Multipass optimization (recommended)
    
        -P<file>    pre-includes a file
        -D<macro>[=<value>] pre-defines a macro
        -U<macro>   undefines a macro
        -X<format>  specifies error reporting format (gnu or vc)
        -w+foo      enables warning foo (equiv. -Wfoo)
        -w-foo      disable warning foo (equiv. -Wno-foo)
    
    --prefix,--postfix
      this options prepend or append the given argument to all
      extern and global variables
    
    Warnings:
        error                   treat warnings as errors (default off)
        macro-params            macro calls with wrong parameter count (default on)
        macro-selfref           cyclic macro references (default off)
        macro-defaults          macros with more default than optional parameters (default on)
        orphan-labels           labels alone on lines without trailing `:' (default on)
        number-overflow         numeric constant does not fit (default on)
        gnu-elf-extensions      using 8- or 16-bit relocation in ELF32, a GNU extension (default off)
        float-overflow          floating point overflow (default on)
        float-denorm            floating point denormal (default off)
        float-underflow         floating point underflow (default off)
        float-toolong           too many digits in floating-point number (default on)
        user                    %warning directives (default on)
    
    response files should contain command line parameters, one per line.
    
    valid output formats for -f are (`*' denotes default):
      * bin       flat-form binary files (e.g. DOS .COM, .SYS)
        ith       Intel hex
        srec      Motorola S-records
        aout      Linux a.out object files
        aoutb     NetBSD/FreeBSD a.out object files
        coff      COFF (i386) object files (e.g. DJGPP for DOS)
        elf32     ELF32 (i386) object files (e.g. Linux)
        elf64     ELF64 (x86_64) object files (e.g. Linux)
        as86      Linux as86 (bin86 version 0.3) object files
        obj       MS-DOS 16-bit/32-bit OMF object files
        win32     Microsoft Win32 (i386) object files
        win64     Microsoft Win64 (x86-64) object files
        rdf       Relocatable Dynamic Object File Format v2.0
        ieee      IEEE-695 (LADsoft variant) object file format
        macho32   NeXTstep/OpenStep/Rhapsody/Darwin/MacOS X (i386) object files
        macho64   NeXTstep/OpenStep/Rhapsody/Darwin/MacOS X (x86_64) object files
        dbg       Trace of all info passed to output stage
        elf       ELF (short name for ELF32)
        macho     MACHO (short name for MACHO32)
        win       WIN (short name for WIN32)
    

    So how would we replace the Win32 functions? I only really need the following functions:

    ALLOC_MEMORY: (could getaway with BSS)
    PRINT_STRING: (outputs a zero terminated string to terminal)
    GET_FILE_LENGTH: (could live without - just gets size in bytes of filename)
    LOAD_FILE: (loads x BYTES of file)
    SAVE_FILE: (saves x BYTES of to a file)
    GET_COMMAND_LINE: (gets pointer to command line string)
  • edited November 2010
    ASM wrote: »
    I just downloaded the latest nasm and it does do macho32/macho64.

    So there's hope!
    So how would we replace the Win32 functions? I only really need the following functions:

    ALLOC_MEMORY: (could getaway with BSS)
    PRINT_STRING: (outputs a zero terminated string to terminal)
    GET_FILE_LENGTH: (could live without - just gets size in bytes of filename)
    LOAD_FILE: (loads x BYTES of file)
    SAVE_FILE: (saves x BYTES of to a file)
    GET_COMMAND_LINE: (gets pointer to command line string)

    You provide a macho format main.obj and then someone else can grab it and write a library that implements the Win32 functions that you use. Link that library against main.obj and hopefully a working executable is the result. For example, in an earlier post you said you used GlobalAlloc to allocate memory. Something like this would probably work:
    void* GlobalAlloc(unsigned int uFlags, size_t dwBytes)
    {
      // Can probably ignore uFlags.
      return malloc(dwBytes * sizeof(char));
    }
    

    None of the other Win32 functions you listed look especially challenging to implement so this shouldn't be too hard...
Sign In or Register to comment.