Syscalls Explained: The Gateway to Bypassing User-Mode Hooks (Part 1)

After covering the basics of system calls in the first part, it's time to get practical. In this article, we'll show you how to bypass User-Mode hooks using direct syscalls.

Our Test Setup

First, we'll demonstrate what a normal call to NtAllocateVirtualMemory looks like — with one twist: we've placed a Detour Hook on this function beforehand.

After that, we'll call the same function using a direct syscall to bypass the hook.

The following diagram illustrates the planned flow:

As you can see, the call to VirtualAlloc (which internally calls NtAllocateVirtualMemory) gets redirected to our hook function HkNtAllocateVirtualMemory. The hook simply prints "Hook called!" — it's only there so we can verify whether our direct syscall actually bypasses the hook.

The Hook

NTSTATUS NTAPI HkNtAllocateVirtualMemory(
    HANDLE ProcessHandle,
    PVOID* BaseAddress,
    ULONG_PTR ZeroBits,
    PSIZE_T RegionSize,
    ULONG AllocationType,
    ULONG Protect)
{
    std::cout << "Hook called!\n";
    
    return reinterpret_cast<tNtAllocateVirtualMemory>(oNtAllocateVirtualMemory)(
        ProcessHandle,
        BaseAddress,
        ZeroBits,
        RegionSize,
        AllocationType,
        Protect
    );
}

Now let's call the function normally:

auto ret = reinterpret_cast<tNtAllocateVirtualMemory>(NtAllocateVirtualMemoryAddr)(
    GetCurrentProcess(), 
    &baseAddress, 
    0, 
    &size, 
    MEM_COMMIT | MEM_RESERVE, 
    PAGE_READWRITE
);

We should see the following output:

The hook was triggered as expected.

---

Direct Syscalls

Direct Syscalls do exactly what the name suggests: they invoke the syscall directly by embedding the assembly instruction in our own program. This completely bypasses ntdll.dll.

Assembly Stub

First, we create a simple stub in an .asm file:

.code

MyNtAllocateVirtualMemory PROC
    mov r10, rcx          ; x64 Fast Call Convention
    mov eax, 24           ; Syscall number for NtAllocateVirtualMemory
    syscall
    ret
MyNtAllocateVirtualMemory ENDP

END

Now we can call the stub like a regular function:

ret = MyNtAllocateVirtualMemory(
    NtCurrentProcess(),
    &baseAddress,
    0,
    &size,
    MEM_COMMIT | MEM_RESERVE,
    PAGE_READWRITE
);

And voilà — the hook is not triggered:

---

Problems with Direct Syscalls

Direct syscalls are very powerful against User-Mode hooks and many EDR/AV solutions. However, they still have some drawbacks:

1. Return Address on the Stack

With a direct syscall, the return address on the thread's call stack points directly into our own code. This is a clear red flag for experienced analysts and modern EDR systems.

2. Syscall Instruction in the Binary

The syscall instruction lives directly in your .exe/.dll. This makes it trivial to statically detect that direct syscalls are being used.

3. Syscall Numbers Change

Syscall numbers are not consistent across Windows versions. Hardcoded values will only work on specific builds.

---

The End

As usual, you can find the code on my github.