Wednesday, July 22, 2015

Solving "unresolved external symbol ___report_rangecheckfailure" Visual Studio linker errors

Let's say you import a library from Visual Studio 2012 or later into your project in an older version of Visual Studio (e.g. Visual Studio 2008 or Visual Studio 2010) but now get linker errors:

error LNK2019: unresolved external symbol ___report_rangecheckfailure referenced in function ...
error LNK2001: unresolved external symbol ___report_rangecheckfailure ...

Sad day. Especially since you don't really get a say in how that library is being built. Your options are:

  1. Upgrade your version of Visual Studio. That includes going through the whole project upgrade cycle. We know how well that usually goes.
  2. Recompile the library yourself. Sad day turns into sad week.
  3. Hack it.
The function __report_rangecheckfailure() is called when the /GS compiler option is used. The option enabled buffer overflow security cookie checking, which, in this day and age, is a good option to have enabled. Unfortunately, that causes problems with older versions of Visual Studio. Let's take a look at what the function does - the source code from 'VC\crt\src\gs_report.c' has this code:

// Declare stub for rangecheckfailure, since these occur often enough that the code bloat
// of setting up the parameters hurts performance
__declspec(noreturn) void __cdecl __report_rangecheckfailure(void)
{
    __report_securityfailure(FAST_FAIL_RANGE_CHECK_FAILURE);
}
Hmm...not really helpful since it calls another function. However, that function contains this very interesting comment after a lot of inline assembler and macros:

    /*
     * Raise the security failure by passing it to the unhandled exception
     * filter and then terminate the process.
     */
So, knowing this normally triggers an unhandled exception and exits the process, we can hack it:

__declspec(noreturn) void __cdecl __report_rangecheckfailure(void)
{
    ::ExitProcess(1);
}
I'm not sure whether to congratulate myself on this evil hack or cry. I think I'll do a little of both. You're welcome.

Oh, and if you work on the Microsoft Visual Studio development team, please develop a compatibility library that implements stuff correctly for older Visual Studio environments. Doesn't have to go back to 1995 VS6, but something reasonable like a 10 year window that addresses issues like these.

3 comments:

  1. a nice idea :)
    though sadly is not working in here. Strange: the call to that thing lies into a .lib file , nonetheless it should work...

    ReplyDelete
  2. I can't do that, it says

    #if defined (_X86_)
    __declspec(noreturn) void __cdecl __report_gsfailure(void)
    #else /* defined (_X86_) */
    __declspec(noreturn) void __cdecl __report_gsfailure(ULONGLONG StackCookie)
    #endif /* defined (_X86_) */

    ReplyDelete
  3. Given that two comments showed up over a short time frame related to this issue, I'm going to guess that the Cumulative Update released on September 14, 2016 (14.0.25431.01) had something to do with it. The original blog post was relevant at the time it was written and the fix did work as well. It was option 3 after all - a hack that could potentially break.

    ReplyDelete