Symbol mismatch can occur when you use a binary compiled on one machine and attempt to run it on a different machine, especially if the two machines have different configurations or architectures. Here’s why this can happen:
Library Dependencies: Binaries often rely on dynamic link libraries (shared libraries) or other system libraries. If the target machine doesn’t have the same versions of these libraries or they are missing altogether, you can encounter symbol mismatch errors.
Architecture Differences: If the two machines have different CPU architectures (e.g., x86 vs. ARM), binaries compiled for one architecture may not run on the other. This is a fundamental incompatibility.
Operating System Differences: Even if two machines have the same architecture, they may have different operating systems with different system calls and ABI (Application Binary Interface) specifications. This can lead to symbol mismatches.
Compiler and Compiler Options: The compiler used to build the binary can affect symbol compatibility. Different compiler versions or options might generate different symbol names or behaviors.
Bitness: Some operating systems and architectures have both 32-bit and 64-bit versions. Trying to run a binary compiled for one bitness on a machine of a different bitness can result in symbol mismatches.
To avoid symbol mismatch issues when moving binaries between machines:
Use Static Linking: Consider statically linking libraries into your binary when compiling. This bundles the necessary libraries into the binary, reducing dependencies on external libraries.
Build on the Target Machine: Whenever possible, compile your code on the machine where you intend to run it. This ensures that the binary is built with the correct dependencies and configurations.
Cross-Compilation: If you must build on one machine and run on another, use cross-compilation tools to generate binaries specifically tailored for the target machine’s architecture and operating system.
Package Managers: If you’re working with package managers (e.g., apt, yum, brew), use them to manage library dependencies and ensure compatibility between systems.
Containerization: Consider using containerization technologies like Docker to package your application along with its dependencies, ensuring portability across different environments.
When you generate a core dump file (usually named “core”) on one machine (Machine B in your scenario) and attempt to debug it using GDB on the same machine, symbol mismatch should not be a significant issue. Here’s why:
Binary Compatibility: The core dump file contains information about the state of the program at the moment it crashed or was interrupted. This includes the memory addresses, registers, and symbol names relevant to the binary that generated the core dump. Since you’re using GDB on the same machine where the binary was executed (Machine B), there should be no symbol mismatch problems related to the architecture or libraries of Machine A.
GDB Compatibility: GDB is designed to work with core dump files generated by the same binary or a compatible binary. It will use the debugging information (symbols) embedded in the binary to analyze the core dump. As long as the binary and the core dump are compatible in terms of architecture, compiler options, and library versions, you should be able to use GDB without significant issues.
Symbol Resolution: GDB uses the symbol information present in the binary (if it was compiled with debugging symbols) to resolve symbols during debugging. It doesn’t rely on external symbol files or libraries when debugging a core dump on the same machine where the program was running.
# 执行程序,程序会出现段错误而退出,并输出相关的错误信息 # 如果编译时没有加-g参数,输出的信息就会少很多(比如行号和具体的代码就没有了) (gdb) run
Starting program: xxx/segment_fault
Program received signal SIGSEGV, Segmentation fault. 0x000000000040051d in main () at segment_fault.cpp:3 3 *num = 100;
3.1.1 set args
The set args command in GDB allows you to specify or change the command-line arguments for the program you are debugging during an active GDB session. This can be particularly useful if you want to test your program with different arguments without restarting GDB.
1
set args [arguments]
Examples:
set args -l a -C abc
set args --gtest_filter=TestXxx.caseX
3.1.2 --args
The --args option in GDB allows you to specify the program and its arguments directly from the command line when starting GDB. This can be very convenient for debugging programs that require command-line arguments.
# 在行号为8的位置打断点 (gdb) break 8 Breakpoint 1 at 0x400848: file set_break.cpp, line 8.
# 在行号为10的位置打断点 (gdb) break set_break.cpp:10 Breakpoint 2 at 0x400864: file set_break.cpp, line 10.
# 在行号为12的位置打断点 (gdb) break 12 Breakpoint 3 at 0x40086b: file set_break.cpp, line 12.
# 在行号为4的位置打断点 (gdb) break 4 Breakpoint 4 at 0x400821: file set_break.cpp, line 4.
# 在行号为17的位置打断点 (gdb) break 17 Breakpoint 5 at 0x400881: file set_break.cpp, line 17.
# 在行号为20的位置打断点 (gdb) break 20 Breakpoint 6 at 0x40089a: file set_break.cpp, line 20.
# 在函数funcA处打断点,发现该断点已经重复了 (gdb) break funcA Note: breakpoint 4 also set at pc 0x400821. Breakpoint 7 at 0x400821: file set_break.cpp, line 4.
# 查看所有断点 (gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y 0x0000000000400848 in main() at set_break.cpp:8 2 breakpoint keep y 0x0000000000400864 in main() at set_break.cpp:10 3 breakpoint keep y 0x000000000040086b in main() at set_break.cpp:12 4 breakpoint keep y 0x0000000000400821 in funcA() at set_break.cpp:4 5 breakpoint keep y 0x0000000000400881 in main() at set_break.cpp:17 6 breakpoint keep y 0x000000000040089a in main() at set_break.cpp:20 7 breakpoint keep y 0x0000000000400821 in funcA() at set_break.cpp:4
# 执行命令run开始运行程序,发现现在程序卡在了行号为8的位置 (gdb) run Starting program: xxx/set_break
Breakpoint 1, main () at set_break.cpp:8 8 std::cout << "hello world" << std::endl;
3.4 Debugging
continue:继续运行直至程序结束或者遇到下一个断点
step:源码级别的单步调试,会进入方法,另一种说法是step into
next:源码级别的单步调试,不会进入方法,将方法调用视为一步,另一种说法是step over
stepi:指令级别的单步调试,会进入方法,另一种说法是step into
nexti:指令级别的单步调试,不会进入方法,将方法调用视为一步,另一种说法是step over
until:退出循环
finish:结束当前函数的执行
display <variable>:跟踪查看某个变量,每次停下来都显示它的值
undisplay <display_id>:取消跟踪
watch:设置观察点。当被设置观察点的变量发生修改时,打印显示
thread <id>:切换调试的线程为指定线程
up [<n>]:沿着栈往上走一层或n层
down [<n>]:沿着栈网下走一层或n层
frame:显示当前的栈信息,包括当前的源码
frame <n>:跳转到栈的指定层
attach <pid>:重新连接到某个进程
3.5 Display Information
bt、backtrace、where:查看当前调用堆栈
bt 3:最上面3层
bt -3:最下面3层
disassemble:查看当前的汇编指令
disassemble:当前函数的汇编指令
disassemble <function>:指定函数的汇编指令
set disassembly-flavor intel:汇编风格指定为Intel Syntax
set disassembly-flavor att:汇编风格指定为AT&T Syntax,该风格为默认风格
list:查看源码
list:紧接着上一次的输出,继续输出后10行源码
list -:紧接着上一次的输出,继续输出前10行源码
list <linenumber>:输出当前文件指定行号开始的10行源码
list <linenumber>, <end_linenumber>:输出指定行号区间的源码
list <function>:输出指定函数的10行源码
list <filename:linenum>:输出指定文件指定行号开始的10行源码
list <filename:function>:输出指定文件指定函数的10行源码
set substitute-path /root/starrocks /other/path/starrocks:修改源码索引路径。当二进制在A机器或者docker内编译,但是在机器B上分析core文件,源码路径通常是对不上的,因此需要用这个命令来调整一下
info用于查看各种调试相关的信息
info break:查看断点
info reg:查看寄存器
info all-reg:查看所有寄存器,包括浮点寄存器和向量寄存器
info stack:查看堆栈
info thread:查看线程
info locals:查看本地变量
info args:查看参数
info symbol <address>:查看指定内存地址所对应的符号信息(如果有的话)
print:用于查看变量
print <variable>
print <variable>.<field>
print (<type>)*<address>:查看地址指向的对象,需要转型
print *(<type>*)<address>:查看地址指向的对象,需要转型
print <array>[0]@5:查看从下标0开始,长度为5的子集
查看、设置属性:show print <property>、set print <property> on/off,下面列出几个常用的属性名(全部属性可以通过show print [tab][tab]或者help show print来查看)
<signal>: The name or number of the signal (e.g., SIGINT, SIGSEGV)
<action>: One or more actions to specify how GDB should handle the signal. The actions can include:
nostop: GDB should not stop the program when this signal is received.
stop: GDB should stop the program when this signal is received.
noignore: GDB should not ignore the signal (default action for most signals).
ignore: GDB should ignore the signal.
noprint: GDB should not print a message when the program receives this signal.
print: GDB should print a message when the program receives this signal.
3.9 Tips
3.9.1 Redirect source file path
The set substitute-path command is used in GDB (GNU Debugger) to remap source paths. This is useful when the source code was compiled on one machine with a different directory structure and you need to debug it on another machine where the directory structure is different.
1
(gdb) set substitute-path <original-path> <new-path>
3.9.2 Redirect Thread Info to File
1 2 3 4 5
(gdb) set pagination off (gdb) set logging file /tmp/threads.txt (gdb) set logging on (gdb) info threads (gdb) set logging off
3.9.3 Redirect Thread Stack to File
1 2 3 4 5
(gdb) set pagination off (gdb) set logging file /tmp/threads.txt (gdb) set logging on (gdb) thread apply all bt (gdb) set logging off
GNU gdb (GDB) Red Hat Enterprise Linux 10.2-6.el7 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from main...
warning: Can't open file /run/rosetta/rosetta during file-backed mapping note processing
warning: core file may not match specified executable file. [New LWP 17530]
warning: Selected architecture i386:x86-64 is not compatible with reported target architecture aarch64
warning: Unexpected size of section `.reg/17530' in core file.
warning: Unexpected size of section `.reg2/17530' in core file. Core was generated by `/run/rosetta/rosetta ./main ./main'. Program terminated with signal SIGABRT, Aborted.
warning: Unexpected size of section `.reg/17530' in core file.
warning: Unexpected size of section `.reg2/17530' in core file. #0 0x0000effff7dfba50 in ?? ()
And there’s a mechanism called Rosetta:
This is a software bridge that allows applications compiled for one instruction set architecture (such as Intel x86) to run on a different architecture (like Apple’s ARM-based processors). Apple has used two versions: Rosetta for the transition from PowerPC to Intel processors, and Rosetta 2 for the transition from Intel to Apple Silicon.
-p <boolean> ( --pass <boolean> ): Whether or not the signal should be passed to the process.
-s <boolean> ( --stop <boolean> ): Whether or not the process should be stopped if the signal is received.
-n <boolean> ( --notify <boolean> ): Whether or not the debugger should notify the user if the signal is received.
When debugging process with JNI, you may receive many interruptions from libjvm.so, like stop reason = signal SIGSEGV: address access protected, which is truly annoying, so you can use this to ignore them
1 2 3
(lldb) process handle -p true -s false -n true SIGSEGV
(lldb) process handle
4.4 How to print env
For environment variables set up before starting program, we can check them by /proc/<pid>/environ. But for environment variables set up at runtime, we can check them using the following approach: