The Breuery

10/10/2020

I had a lot of problems betting win32/FFI working. This file is a record of my journey.

I've been working on my LispHack project. Nethack is a classic video game with a certain cult following. It's old, so the graphics are all text only. A little @ journeys through random levels, fighting monsters of various letters and finding treasureslike . and $ if your lucky. I am stalled on my Lisp implementation.

For my mom, Lisp is a very old computer language that the very smart cool kids like. Normal computer programs let you add the numbers in a list or insert a number at the proper place. Lips takes that a flying step further and treats its own programs as a list, thus you can easily change the program itself as its running (or nearly). If you need to sort a program (what?) or insert a number into it (sure...) Lisp has tools that embrace that. I often think that studying Lisp will either lead to enlightenment or insanity; I know I've not reached enlightenment and me staying up hours after my bedtime to solve a puzzle is not looking good for the insanity.

Lisp is old and strange and has a strange syntax to go with it. In C++ which I most regularly use, I have several types of parenthesis that do different jobs. () {} [] <>. I know it looks crazy to you, but each has its place. For math and the start of certain thoughts, we use (). For building data or programs we use {}. For finding items in lists we use []. And for taking a generic list and asking for an optimized version for names or distances only, we use <>. At a glance I can see what the different parts of the program are doing.

Lisp has only (). And the verbs are in a different order. And there's something strange going on with the quotations. (something `(,1 '2 ,3 (doit 1 2) ())) is valid Lisp or very nearly and the monotonous parenthesis hurt my head.

Anyway, back to my problem.

My variant of Lisp has two display modes. The black-and-white pile of text that goes streaming past called the console, and a sliiiightly more advanced version called SCREEN:MAKE-WINDOW. The console has no ability to do colors or to move the draw point around the screen and while SCREEN can move the draw point, it has a hideous white on blue appearance, with the aded insult that I can change the color of the text from white to slightly brighter white.

Fully colored, fully positioned text can be done. I have done it several times. A game I love, "Dwarf Fortress" is in the same ASCII text genre, and their implementation is to run a fully modern 3D OpenGL renderer, and then only render colored text. I really hope I don't have to go that far. If I can patch into windows advanced console coloring tools, I can have everything. If.

I am using GNU Common Lisp CLISP.EXE, version 2006? I believe that there is a GNU Common Lisp GCL.exe and a GNU Common Lisp CLISP.EXE, so that adds to the confusion. "Common" Lisp is not common, everyone has made their own implementations which are Turing compatible with each other, but not linguistically compatible. /snark.

  i i i i i i i       ooooo    o        ooooooo   ooooo   ooooo
  I I I I I I I      8     8   8           8     8     o  8    8
  I  \ `+' /  I      8         8           8     8        8    8
   \  `-+-'  /       8         8           8      ooooo   8oooo
    `-__|__-'        8         8           8           8  8
        |            8     o   8           8     o     8  8
  ------+------       ooooo    8oooooo  ooo8ooo   ooooo   8

Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2006

Unfortunately, not only is Lisp a wildly varied community of separate languages, the FFI ecosystem is too. While the Lisp programmers are the coolest programmers, the stolid coders who are getting anything done are working in C. Most languages allow some method of reaching out of their own language and asking C to finish some task for them. In Lisp, any of the Foreign Function Interfaces (calling some's else's code that wasn't written in Lisp) claim to do the job. Unfortunately, there are several. FFI, CFFI, UFFI, FLI to name a few. Each of these methods will let me call a C/Windows function, but the code required is different, so it's hard to search for help. CLISP.EXE has a module called FFI, so that's what I'm using.

test.lisp

The fifth or more problem I hit is that Lisp is usually case-insensitive so BROWNCOW and BrOwNcOw mean the same thing. Windows code on the other hand is very picky about the spelling of GetCursorPos. I copied win32.lisp's approach of creating a module and marking it as Lisp-Modern.

(defpackage "TTest"
  (:modern t)
  (:use "COMMON-LISP" "FFI"))

(require "../lisp/bindings/win32/win32.lisp")

(ffi:default-foreign-language :stdc-stdcall) ; WINAPI means __stdcall
(eval-when (compile) (setq ffi:*foreign-guard* t))

(in-package "TTest")

I was suprised that IN-PACKAGE was required after defining the module. I guess that's the way Lisp works.

(FFI:DEF-C-STRUCT POINT
	(x ffi:sint32)
	(y ffi:sint32))

I had a lot of problems referencing the win32 types like LONG or DWORD from my file. I eventually recreated some of those types from teh FFI.

(ffi:def-call-out GetCursorPos (:library win32:user32)
	(:arguments (lpBuffer (ffi:c-ptr POINT) :out :alloca) )
	(:return-type win32:dword))

(multiple-value-bind (success pPoint)
	(GetCursorPos )

	(format t "GetCursorPos: ~a ~a~%" success pPoint))

My version of FFI uses def-call-out to find a function in a different language, then build a wrapper so it can be called more easily from inside Lisp. :out and :alloca are very useful for certain Windows functions because C tends to accept outputs as inputs "Tell me where you want the answer, and I'll whisper the answer there". Lisp doesn't need that noise and wants both answers returned as outputs. "Everything's fine and the cursor is at this location." If I just wanted to do the work, (GetCursorPos) is sufficient. Because there's two answers hidden in the output, I'm using multiple-value-bind to name them and separate them.

I hit a lot of errors, so I added (format t "" asdfasdf) everywhere. That prints some text on teh screen, telling me that everything worked or didn't.

(format t "GetStdHandle: ~a~%" (win32:GetStdHandle win32:STD_INPUT_HANDLE))

(format t "GetConsoleWindow: ~a~%" (win32:GetConsoleWindow))

(ffi:def-call-out AllocConsole (:library win32:kernel32)
	(:arguments )
	(:return-type ffi:c-pointer))

(FFI:WITH-C-VAR (consoleh 'ffi:c-pointer (AllocConsole)) (
	format t "AllocConsole: ~a ~a~%" consoleh (win32:GetLastError)))

I was having errors, so I tried several approaches to get access to the output window. win32:STD_INPUT_HANDLE is the name of the keyboard (kinda: input handle on the console) That's a typo and the cause of some of my errors. It should be win32:STD_OUTPUT_HANDLE. (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) is the correct function that takes the name of the output console and returns access to read and write text there. Unfortunately, Windows likes opaque "handles" which give access but don't tell you what they are are what they do, so the handle to a cat, the handle to a cucumber, and the handle to a chainsaw all look the same. :-( win32:GetConsoleWindow returns access to the GUI Windows frame around the console - the title bar and the close buttons and such. Not the text inside. While fighting with a bug, I got confused between the handle to the output console and the handle to the block of text inside the output console. There's no way to ask windows for the current block of text (ScreenBuffer) but you can create a new one and force Windows to use that. AllocConsole lets me create a new one. I think with FFI, that WITH-C-VAR is usually not needed, but I was getting very confused. FFI:WITH-SCREEN-VAR creates a space to work that's ready to be sent into C. Regular Lisp variables are allocated wrong/in the wrong place.

(multiple-value-bind (success pPoint)
	(Win32:GetConsoleScreenBufferInfo (win32:GetStdHandle win32:STD_OUTPUT_HANDLE))

	(format t "GetConsoleScreenBufferInfo: ~a ~a ~a~%" success pPoint (win32:GetLastError)))

One of my big problems is packaging structs or blocks of data up so I can send them to the Windows C code. This left me stuck between the rock of not knowing if I'm using FFI correctly, and the hard place of not knowing if the ordinarily grumpy and pedantic Win32 code was refusing my calls.

GetConsoleScreenBufferInfo was written for me in the existing library. I could print the contents, but the contents were all 0. It took me a while to realize that I had STD_INPUT_HANDLE and STD_OUTPUT_HANDLE mixed up.

(ffi:def-call-out GetConsoleTitleA (:library WIN32:kernel32)
	(:documentation "https://docs.microsoft.com/en-us/windows/console/getconsoletitle")
	(:arguments	(lpBuffer (ffi:c-ptr (ffi:c-array-max ffi:char 256)) :out)
				(nNumberOfCharsToRead win32:dword))
	(:return-type win32:dword))

(setf buffer (FFI:ALLOCATE-DEEP 'FFI:CHARACTER "ABCDEF" :count 14) )

(multiple-value-bind (success lpBuffer)
;; :OUT or :IN-OUT parameters are returned via multiple values

	(GetConsoleTitleA 12)

	(format t " ~a ~s ~%" success (map 'string #'code-char lpBuffer)))

Because I was having trouble with sending large blocks of data and I also needed to figure out text conversion from C strings to Lisp strings, I tried grabbing the text off the titlebar. Again, I'm not certain I need FFI:ALLOCATE-DEEP to create a C compatible memory space to work, FFI:DEF-CALL-OUT does that work for me.

So the working version calls (GetConsoleTitleA 12) - Windows wants to know how long of text I'm expecting so I lied and said 12. Above DEF-CALL-OUT was told to expect 256 characters, so it's fine. It's fine. Then MULTIPLE-VALUE-BIND grabs both the success response and the "tell me where to hide the answer" text I requested. Unfortunately the data is coming back as a list of numbers (Computers love numbers and Lips loves lists.) so I have to use MAP to convert a list of numbers into the text string that represents.

	(win32:WriteConsoleA (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) (FFI:ALLOCATE-DEEP 'ffi:CHARACTER "The Quick Brown Fox Jumped Over The Lazy Dogs.
	" :count 50)  47 '())

Maybe that FFI:ALLOCATE-DEEP was needed to convert from Lisp strings into C strings. I'll be working on this program further when I've finished this diary. (format t "asdf") will send some text to the screen in a happy Lisp fashion. WriteConsoleA does exactly the same thing but via the Windows functions I want to be using. The effect is silly, but I'm slowly proving out that I've got the FFI registrations correct and that the endlessly picky Win32 system is responding correctly. In C, we use NULL to mean "don't know, don't care" or "absolutely nothing at the moment." Lisp uses the eternally empty list '() and FFI converts between the two.

; BOOL WINAPI (
  ; _In_ HANDLE hConsoleOutput,
  ; _In_ WORD   wAttributes
; );
(ffi:def-call-out SetConsoleTextAttribute (:library WIN32:kernel32)
	(:documentation "")
	(:arguments	(handle ffi:c-pointer)
				(wAttributes win32:word))
	(:return-type win32:dword))



(SetConsoleTextAttribute (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) #x1)
(format t "Blue? ")
(SetConsoleTextAttribute (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) #x2)
(format t "Green? ")
(SetConsoleTextAttribute (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) #x4)
(format t "Red? ")
(SetConsoleTextAttribute (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) #x7)
(format t "~%")

Red? Green? Red?

Hallelujah! Hallelujah! HallelujahHallelujahHallelujah! Ha-le-lu-jah!! I think it was at this point I discovered the INPUT versus OUTPUT mistake I'd been making for hours. I was able to get the colors to change. #x7 is how Lisp writes the binary number 0111, which means white but not very bright for this color system.

(ffi:def-call-out SetConsoleCursorPosition (:library WIN32:kernel32)
	(:documentation "")
	(:arguments	(handle ffi:c-pointer)
				(position (ffi:c-pointer win32:COORD)))
	(:return-type win32:dword))

(multiple-value-bind (success)
	(SetConsoleCursorPosition (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) (win32:make-COORD :|x| 10 :|y| 280)) 

	(format t "SetConsoleCursorPosition: ~a ~a ~a~%" (win32:make-COORD :|x| 10 :|y| 280) success (win32:GetLastError))
)

The above code is wrong.

Since I had colored text, my final goal was text written at any arbitrary spot, not just flowing from top to bottom. SetConsoleCursorPosition should let me do that. This failed for sevaral ours more and I went digging for alternate ways to get access to the screen. Once I had colors working, I knew I was close, but I couldn't figure out why the position didn't move. Also note the gross :|x| noise. That clause says there is some data of type COORD, which has an :X part and a :Y part, but I had to insert the lines to get it to compile. Anyway, for the longest time the text didn't move.

I'm still confused about the difference between FFI:C-PTR and FFI:C-POINTER. They're both pronounced the same and do almost the same thing. I think one might be about allocation - I'll tell the computer when I'm done with this and don't forget it before then. I don't know which is which yet.

(ffi:def-call-out CreateFileA (:library WIN32:kernel32)
	(:documentation "")
	(:arguments	(lpFileName ffi:c-string)
				(dwDesiredAccess win32:dword)
				(dwShareMode win32:dword)
				(lpSecurityAttributes ffi:c-pointer)
				(dwCreationDisposition win32:dword)
				(dwFlagsAndAttributes win32:dword)
				(hTemplateFile win32:handle))
	(:return-type win32:handle))

(setf consoleHandle ( CreateFileA "CONOUT$" #x60000000 #x2 '() 3 0 '() ));
 
(format t "CreateFileA: ~a ~a~%" consoleHandle (win32:GetLastError))

I looked up alternate ways to get access to the console like the global pseudo-file "CONOUT$". This didn't work, but you can see how complicated the FFI:DEF-CALL-OUT registration is. I was getting more comfortable with the registration process.

; BOOL WINAPI SetConsoleCursorPosition(
  ; _In_ HANDLE hConsoleOutput,
  ; _In_ COORD  dwCursorPosition
; );

(ffi:def-call-out SetConsoleCursorPosition (:library WIN32:kernel32)
	(:documentation "")
	(:arguments	(handle ffi:c-pointer)
				(position win32:COORD))
	(:return-type win32:dword))

And in a StackOverflow help post I notice the difference. COORD is small. It is built as a block of data with an X horizontal part and a Y vertical part, but those numbers are small so the whole COORD is small. Most windows functions ask for a pointer to data - set up the data somewhere and tell the function where to find it. It's like getting a letter with the name and address of a person in it. This Windows function takes the entire structure packed into a parameter. It's like getting that entire person mailed to you in the post. Most Windows fucntions don't do this. Most Windows functions can't do this. Adults don't fit in a flat-rate box.

(format t "Here? ")

(SetConsoleCursorPosition (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) (win32:make-COORD :|x| 10 :|y| 280))

(SetConsoleTextAttribute (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) #xF)

(win32:WriteConsoleA (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) (FFI:ALLOCATE-DEEP 'ffi:CHARACTER "Over here?.
" :count 50)  10 '())
		
(SetConsoleTextAttribute (win32:GetStdHandle win32:STD_OUTPUT_HANDLE) #x7)

(format t "Or down here? ")

Over Here? Or down here? Here?

Halle-Gloria-tastic! #xF means bright white. #x7 means grey/medium white. In the code above I print "Here? Over here? Or Down Here?", but SetConsoleCursorPosition moves the current write position around as I'm drawing the text.

I couldn't get my wife as excited by putting colored text on the screen as I am, but like most things I do, the text is the least important part and the knowledge I've gained is the important part. The FFI examples I had only moved simple numbers around and not structs of data. I can do both now. Win32 functions are terribly picky, but I've got half a dozen of them obeying me now. I know where to get access to Windows functions. I can convert Lisp strings to C strings and back. (This is fundamental because both languages are picky and incompatible.) I have C-POINTER/C-PTR doing something. I'm still worried that part's only half right.

I'm a programmer. This file may only write a pile of text onto the screen, but it's used a lot of cool and complicated technology to do it. :-)

10-11-2020

I worked some more on this, using my new knowlege to write FFI hooks for each of the windows console functions. I needed to call GetConsoleScreenInfo and then examine info.size.x to see how large the screen was. I could print "info" and see all its members, but attempting to write (let (size (info.size)) ... ) showed that size was nil. Again I dig for an hour. Is FFI not working? This is the first time I've examine a field in a sub-structure. One thing that's super confusing - Lisp is erratic about case sensitivity. It's an old language and thinks of itself in ALL CAPS. To be friendly, it treats both lower-case and UPPER-CASE as the same. Internally, though it uses UPPER-CASE. The print-to-screen format command however wants to display things in a pretty manner for humans, so it reverses UPPER-CASE into lower-case. And the fastest way to convert is to just flip-flop all the letters. I told you all this to tell you this. Windows uses something called printedCamelCase. The field I am interested in is called dwMaximumWindowSize.X and to be helpful, format prints that as "DWmAXIMUMwINDOWsIZE.x". All in all, I'm super confused as to whether I'm even asking for the right entry.

An hour and a half later... It's spelled (let ((size (info.size))) ...) not (let (size (info.size)) .. ) Silly me.