openwrt开发一个进程，调用curl共享库，崩溃了，gdb了一下，bt显示崩溃在libcurl.so中，但是没有打印函数名，打印的是 ??
应该是共享库被strip了，没有调试信息，
进到curl的编译目录：
$ cd build_dir/target-arm-openwrt-linux-muslgnueabi_musl/curl-7.79.1
搜索一下编译产物，哪些带符号信息：
$ find . -name 'lib*.so*' | xargs file
./ipkg-arm_cortex-a7/libcurl/usr/lib/libcurl.so.4: symbolic link to libcurl.so.4.7.0
./ipkg-arm_cortex-a7/libcurl/usr/lib/libcurl.so.4.7.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, stripped
./.pkgdir/libcurl/usr/lib/libcurl.so.4: symbolic link to libcurl.so.4.7.0
./.pkgdir/libcurl/usr/lib/libcurl.so.4.7.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_i
nfo, not stripped
./ipkg-install/usr/lib/libcurl.so: symbolic link to libcurl.so.4.7.0
./ipkg-install/usr/lib/libcurl.so.4: symbolic link to libcurl.so.4.7.0
./ipkg-install/usr/lib/libcurl.so.4.7.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_i
nfo, not stripped
./lib/.libs/libcurl.so: symbolic link to libcurl.so.4.7.0
./lib/.libs/libcurl.so.4: symbolic link to libcurl.so.4.7.0
./lib/.libs/libcurl.so.4.7.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_i
nfo, not stripped
ls看看大小：
$ find . -name 'lib*.so*' | xargs ls -l
lrwxrwxrwx 1 yuan yuan 16 Mar 11 19:32 ./ipkg-arm_cortex-a7/libcurl/usr/lib/libcurl.so.4 -> libcurl.so.4.7.0
-rwxr-xr-x 1 yuan yuan 305284 Mar 11 19:32 ./ipkg-arm_cortex-a7/libcurl/usr/lib/libcurl.so.4.7.0
lrwxrwxrwx 1 yuan yuan 16 Mar 11 19:32 ./ipkg-install/usr/lib/libcurl.so -> libcurl.so.4.7.0
lrwxrwxrwx 1 yuan yuan 16 Mar 11 19:32 ./ipkg-install/usr/lib/libcurl.so.4 -> libcurl.so.4.7.0
-rwxr-xr-x 1 yuan yuan 372072 Mar 11 19:32 ./ipkg-install/usr/lib/libcurl.so.4.7.0
lrwxrwxrwx 1 yuan yuan 16 Mar 11 19:32 ./lib/.libs/libcurl.so -> libcurl.so.4.7.0
lrwxrwxrwx 1 yuan yuan 16 Mar 11 19:32 ./lib/.libs/libcurl.so.4 -> libcurl.so.4.7.0
-rwxr-xr-x 1 yuan yuan 372072 Mar 11 19:32 ./lib/.libs/libcurl.so.4.7.0
lrwxrwxrwx 1 yuan yuan 16 Mar 11 19:32 ./.pkgdir/libcurl/usr/lib/libcurl.so.4 -> libcurl.so.4.7.0
-rwxr-xr-x 1 yuan yuan 372072 Mar 11 19:32 ./.pkgdir/libcurl/usr/lib/libcurl.so.4.7.0
可以发现 lib/.libs .pkgdir ipkg-install 里面的文件是一样的，带调试信息，ipkg-arm_cortex-a7目录中的不带调试信息。
把带调试信息的共享库上传到路由器替换（bind mount），把带调试信息的共享库拷贝到openwrt的rootfs目录（后面gdb用）
$ cp ./lib/.libs/libcurl.so.4.7.0 /work/tftproot/
$ cd ../root-ipq/
$ cp /work/tftproot/libcurl.so.4.7.0 ./usr/lib/libcurl.so.4.7.0
在路由器上：
$ tftp -g 192.168.10.100 -r libcurl.so.4.7.0
$ mount --bind libcurl.so.4.7.0 /usr/lib/libcurl.so.4.7.0
运行进程，产生core文件，将core文件传到本地，使用gdb调试，
$ arm-openwrt-linux-gdb obj/dlife /work/tftproot/dlife.1710725879.3026.11.core
GNU gdb (GDB) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=arm-openwrt-linux-muslgnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from obj/dlife...done.
Reading symbols from obj/dlife...done.
[New LWP 3026]
warning: Could not load shared library symbols for 14 libraries, e.g. /usr/lib/libfuse.so.2.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `./dlife -v'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x76e92bfc in ?? ()
(gdb) set sysroot /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libfuse.so.2...(no debugging symbols found)
...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libubox.so...(no debugging symbols found)...don
e.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libubus.so...(no debugging symbols found)...don
e.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libuci.so...(no debugging symbols found)...done
.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libssl.so.1.1...(no debugging symbols found)...
done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libcrypto.so.1.1...(no debugging symbols found)
...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libjson-c.so.2...(no debugging symbols foun
d)...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libuuid.so.1...(no debugging symbols found)
...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/liblua.so.5.1.5...(no debugging symbols fou
nd)...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libgcc_s.so.1...(no debugging symbols found)...
done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/ld-musl-arm.so.1...(no debugging symbols found)
...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcares.so.2...(no debugging symbols found
)...done.
Reading symbols from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libnghttp2.so.14...(no debugging symbols fo
und)...done.
(gdb) bt
#0 0x76e92bfc in Curl_strncasecompare () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#1 0x76e9317c in Curl_checkheaders () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#2 0x76e7c0fc in Curl_http_host () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#3 0x76e7d5b4 in Curl_http () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#4 0x76e89d68 in multi_runsingle () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#5 0x76e8a9f8 in multi_socket () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#6 0x76e8aac0 in curl_multi_socket_action () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/usr/lib/libcurl.so.4
#7 0x00015fe4 in on_sock_event (u=0xfec998, events=1) at curl_uloop.c:44
#8 0x76eef5ac in uloop_run_timeout () from /work/ax6000/build_dir/target-arm-openwrt-linux-muslgnueabi_musl/root-ipq/lib/libubox.so
#9 0x0001ebb8 in uloop_run () at /work/ax6000/staging_dir/target-arm-openwrt-linux-muslgnueabi_musl/usr/include/libubox/uloop.h:116
#10 0x0001f7d8 in main (argc=2, argv=0x7e910e94) at main.c:272
可以看到显示了完整的调用栈。总结：
1 找到编译目录中带符号的共享库，
2 将其拷贝到本地openwrt的rootfs目录，替换原来strip后的共享库
3 将其拷贝到路由器上，bind mount的方式，替换原来strip后的共享库
4 重新运行进程，产生core文件
5 gdb set sysroot指定rootfs目录，bt显示调用栈。