Strange crash with WTSOpenServer on Windows 7 (Only in Delphi 2009/2010)

I am troubleshooting a problem with existing code that always worked fine (it's the Terminal Server unit from the Jedi Windows Security Library). After some investigation the problem part has been brought down to a call to WTSOpenServer:

  while true do
  begin
      hServer := WTSOpenServer(PChar('server'));
      WTSCloseServer(hServer);
      hServer := 0;
  end;

After a random (but small) number or runs we get a total app crash which makes it hard to debug. Here are the things I already tried:

• WTSOpenServer does not write to the pServername parameter (like CreateProcessW) (in fact I checked the disassembly and it makes a copy)
• The code runs fine when passing nil as parameter (and thus work with the localmachine).
• When using a remote server, localhost or even dummy as pServerName the result is always crash (On Vista and higher even an invalid servername returns a valid handle as per documentation).
• Tested with both Delphi 2009 and 2010
• The same code runs fine in Visual Studio (c++).

• Checked the disassembly in Visual Studio and made the call the WTSOpenServer in asm from Delphi (and change the Handle type to a pointer like in C):

  hModule := LoadLibrary('wtsapi32.dll');
  if hModule = 0 then
    Exit;
  
  
  WTSOpenServer := GetProcAddress(hModule, 'WTSOpenServerW');
  if WTSOpenServer = nil then
    Exit;
  
  
  while true do
  begin
    asm
      push dword ptr pServerName;
      call dword ptr WTSOpenServer;
      mov [hServer], eax;
    end;
  
  
    hServer := nil;
  end;
  

• Leave out the call to WTSCloseServer

• Test the code on both x64 and x86 version of Windows 7
• Use External Debugger instead of Delphi one (seems to run fine in that case so my guess is that it's some kind of timing/thread/deadlock issue)
• Added AddVectoredExceptionHandler then I see a EXCEPTION_ACCESS_VIOLATION but the stacks seems to be corrupted, EIP is 1 so cannot determine where it happens.

At this point I don't know how to further troubleshoot this or find an explanation.

From stackoverflow Remko

• Try run your application with FastMM in FullDebugMode. It looks more like a bug in your/3rd party-lib code - possible memory overwrite/buffer overflow (moslty like sth. GetMem too small for UnicodeString/String alike operations, and it 'works' but will sooner or later crash/AV).

  I've had several similar situations when migrating big app to D2009, and in most cases it was due to assumption Char=1 byte. Sometimes very strange things happened, but always FullDebugMode helped. Exception was CreateProcessW, but it's know/documented behaviour.

  With FullDebugMode if app overwrite memory, then when you free it, FastMM gives you exception where it was allocated, so easly you can track down this bug. It adds some bytes at begining and end of allocation, so will know if it was overwritten.

  I'm not able to reproduce it with new/empty VCL project, you can try it your self (this loop running for about 5 min):

  uses JwaWtsApi32;
  procedure TForm7.FormCreate(Sender: TObject);
  var
    hServer: DWORD;
  begin
    while true do
    begin
        hServer := WTSOpenServer(PChar('server'));
        WTSCloseServer(hServer);
        hServer := 0;
    end;
  end;
  

  Remko : My project consists of only that code! I even brought it back to a console project with using no units at all. Please note that the crash only occurs when running on Windows 7, when debugging, with Delphi 2009 or Delphi 2010.

  kibab : I'm using also Windows 7 (RC, Ultimate 32-bit) and Delphi 2009 (12.0.3420.21218), and using only one additional to std. new VCL project explicit import - JwaWtsApi32 rev. 820 (updated answer about with this). Running from delphi for 5-10min, and also with step-by-step about 30-40 times by that loop. Are you able reproduce it on another machine with same OS/Delphi? My very extreme thoughts about this: virus(rootkit;)/third party bug in code injected to exe, like madExcept,EurekaLog,JDBG,../unusual default Delphi compiler/linker options that is buggy/sth. wrong with OS, updates, antivir, LSP ...

  Remko : Did you test on RC or RTM?

  kibab : Windows 7 RC "I'm using also Windows 7 (RC, Ultimate 32-bit) ..." (get email from info@simulates...)

  Remko : Although I had disabled my Virusscanner/Firewall I still noticed injections of dll's from it (F-Secure Internet Security 2010). I ignored it before because it worked fine under the older Delphi versions but decided to do an uninstall. And guess what? No more crash, will do some more tests and if this was the problem I will accept your answer.

  kibab : So it was possible buggy LSP library (my last "very extreme thought" -> "sth. wrong with ... LSP" ;) It did overwrite memory... You can use Autoruns (http://technet.microsoft.com/pl-pl/sysinternals/bb963902(en-us).aspx?ppud=4), to list all LSP (WinSock Providers), then you can disable/remove some of them - non Microsoft (you will loose some functionality from eg. F-Secure, but it's buggy, so not worth to use it ;). LSP samples from Windows SDK (from MS) are also buggy (and leaks memory), so mostly like delivered work will be buggy too. I know that F-Secure and E-Set LSP dll's are buggy...
  From kibab