How to find source code lines from appcrashview reports and symbols (Win VS)

Post your tutorials for using applications or performing related tasks here.
Note: Not for "how do I...?" Questions!
Forum rules
Tutorials and Howtos should only relate to developed software, and not to third party applications. e.g.: Don't post a generic Howto for configuring a firewall.
If you have a question how to do something, you should use one of the support boards, not this board. It is meant for people to document and post instructions.
User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

How to find source code lines from appcrashview reports and symbols (Win VS)

Unread post by Moonchild » 2019-07-09, 09:19

While it may seem that I'm waving some magic wand when looking at reports posted by people from appcrashview to magically know exactly where the browser crashes, finding the source code line a crash occurs at from just a report of an appcrash isn't particularly hard, but it does require some work.

Example appcrash:

Code: Select all

Version=1
EventType=APPCRASH
EventTime=xxxxx
ReportType=2
Consent=1
ReportIdentifier=xxxxxxx
IntegratorReportIdentifier=xxxxxxx
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=palemoon.exe
Sig[1].Name=Application Version
Sig[1].Value=4.3.0.7124
Sig[2].Name=Application Timestamp
Sig[2].Value=5d1e4636
Sig[3].Name=Fault Module Name
Sig[3].Value=xul.dll             <- *** FAULTING MODULE ***
Sig[4].Name=Fault Module Version
Sig[4].Value=4.3.0.7124
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=5d1e4821
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=00000000009f4d2a  <- *** CRASH OFFSET ***
DynamicSig[1].Name=OS Version
[more removed that isn't important for this tutorial]
What you'll need:
  • The UXP Source code (check out the tagged release version of the source with git)
    For this tutorial I'll assume you've cloned into C:\source
  • A version of Pale Moon matching the crashing one (e.g. a portable)
  • The binary's debug symbols (available from the archive server shortly after release)
    Unpack the symbol archive with 7-zip, preserving paths, into a folder of your choice.
    Make sure you are using the exact version of the symbols matching the crashing application or you won't be able to use them.
    For this tutorial I'll assume C:\symbols\
  • Visual Studio 2015 CE or later. I'm going by 2015 here because that's what I have installed.
  • A hex-capable calculator
    Windows' calculator will do just fine, just switch to programmer's mode and make sure you're using Hex mode.
Steps:
  1. Check which architecture of Pale Moon is being used, x86 or x64. This is important because addresses will be vastly different depending on architecture.
  2. Launch the same version and architecture of the browser on your system (you can use a portable version for the sake of debugging ease).
  3. With it running, Launch Visual Studio, Go to "Debug" in the menu, then "Attach to process". In the list, select the Pale Moon process you just started, and attach the debugger to it.
  4. First time only: In the Visual Studio IDE, you will have a "modules" panel that lists all the modules loaded in the process you attached to; in this window you can right-click and go to "Symbol settings" to point to where VS can find the program's symbols. I suggest you uncheck the "Microsoft symbol servers" to speed up the debug loading process; we normally don't need to see where exactly in an external module something goes wrong, we're interested in a crash in a Pale Moon module only.
    Symbolsettings.jpg
  5. First time only: In that same window, add the path to the symbols of the crashing module (see appcrashreport module name)
    The most common paths are:
    For xul.dll: C:\symbols]\toolkit\library
    For palemoon.exe: C:\symbols\application\palemoon\app
    Note, you can also browse for the specific symbol file later on when the VS environment asks for it, but it's generally faster to set this up beforehand.
  6. Since modules are loaded at randomized addresses (ASLR), you always have to calculate the exact address in your running version you are debugging yourself with the base address and offset.
    In the modules panel, look up the "Address" column of the crashing module, which has an address range in memory where the module is loaded. Make note of the lowest address in that range and plug it into your calculator.
    module_address.jpg
    module_address.jpg (21.55 KiB) Viewed 3822 times
    In this example, the address is 7FEC84E0000
    Now find the "exception offset" in the reported appcrash, this is the offset of the start address of the module where the crash occurred. Go back to your calculator and ADD this offset to the base address you just found. In this example 7FEC84E0000 + 9f4d2a = 7FEC8ED4D2A
  7. Time to break execution! Press the "pause" button in the visual studio toolbar [||]. This will halt the execution of the program you attached to and should populate a number of panels with information and display a "disassembly" window.
    It may ask you for a source location at this point. Pay close attention to the path in the dialog at the top. There will be a path there that starts with something likely not existing on your system, like d:\mozdev\UXP\, which is why the lookup isn't automatic. the rest of the path, however, will be a subfolder that exists in your C:\source. Browse to that same path in your C:\source tree, and it will automatically close once it finds the correct source file in the path you're browsing to.
    If the disassembly window gives you just a notification that there is nothing to display because all threads were executing in external modules, just press the Play button on the toolbar [|> Continue] and repeat this process until you get a disassembly view.
  8. Now, in the "Address" bar of the disassembly window, enter "0x" and the address you calculated, in this example "0x7FEC8ED4D2A" and press enter. This will jump your view to the actual point in the browser where it crashed to make the appcrash report. VS may once again ask for a source file in a different path, depending on the crash point, see previous point for instructions.
    At this point you have located the exact crash point in the source -- Good job!
    The cursor will be at the exact instruction that crashed. You can make note of this (and/or screenshot it) and include it in your report.
    disassembly.jpg
  9. Now, the disassembly view is usually not very readable, so one last step: Right-click the disassembly view, and click "Go to source code". You will now have the exact source location the crash occurred in source code context. You can copy this section of code into a report and highlight the crashing line.
    In this example:

    Code: Select all

      void SetPostScale(float aXScale, float aYScale)
      {
        if (mPostXScale == aXScale && mPostYScale == aYScale) {
          return;
        }
        MOZ_LAYERS_LOG_IF_SHADOWABLE(this, ("Layer::Mutated(%p) PostScale", this));   <- *** CRASH ***
        mPostXScale = aXScale;
        mPostYScale = aYScale;
        Mutated();
      }
    
Performing these steps yourself to find the crash point in the source will significantly reduce the time I or others have to spend going through this process to look up where something crashes. I know it's not a 2-step thing and will require a bit of effort and learning to get familiar with the tools and process, but if you could do this when reporting crashes, it would be a huge help. Don't be afraid to ask others to help you with this, either. I understand if you simply don't have the time or it's too daunting of a process to go through, but that's why we are a community!
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked