appع

maxkabakov - Fotolia

Problem solve Get help with specific problems with your technologies, process and projects.

Troubleshoot Linux kernel panic with the kdump crash tool

Kernel panic renders your system useless. With an NFS server and kdump crash tool, you can easily perform a root cause analysis and figure out how to bring a system back online.

Kernel panic is a critical issue that manifests as a system freeze. If you're not familiar with what a kernel does, it is the core of an OS. Linux itself is a kernel, which enables developers to create numerous distributions.

A serious enough error at the kernel can cause an event known as kernel panic. This is similar to Window's blue screen of death, but instead of seeing a blue screen, you simply see a log output on a black screen.

Kernel panic can occur due to bad memory, driver crashes, malware or software bugs. To identify the cause of kernel panic, you can use the kdump service to collect crash dumps, perform a root cause analysis and troubleshoot the system.

To get started, you should have two VMs that run CentOS. This tutorial uses CentOS 8 as the Linux distribution for both the Network File System (NFS) server and client.

If you configure the client to send the crash dumps to an NFS share, you can centrally gather and analyze a crash dump without using the system that is affected by kernel panic.

Below are the IP addresses of the NFS server and client. Your addresses may differ depending on your subnet configuration, but both addresses are necessary.

NFS server

192.168.99.1

Client

192.168.99.71

At publication, the kernel version on CentOS 8 is 4.18.0-147.5.1.el8_1.x86_64. To find the distribution's Linux kernel version, use the uname-r command.

Install NFS

After you've and have the IP addresses for the intended NFS server and client, it's time to install everything with the following steps:

  1. Use the yum install -y nfs-utils command to connect to the NFS server and install the NFS package.
  2. Create a directory that will be used by NFS with mkdir /nfs-share.
  3. Edit the /etc/exports file to allow the client to connect to the NFS share. The supplied IP address belongs to the client.
    /nfs-share 192.168.99.71(rw,sync,no_root_squash)
  4. Connect to the client, and create a directory to store the crash dump files; the default for kdump is /var/crash. You can make a directory with the name crash-dump in the root directory with a mkdir/crash-dump prompt.
  5. Mount the new NFS share to the client's file system. The supplied IP address is to the NFS server.
    mount -t nfs 192.168.99.1:/nfs-share /crash-dump
    To verify NFS is correctly set up, run the df -h command. If the setup is successful, your screen should have the following output:

    Successful NFS server installation output

Verify and install kdump and crash tool

Now that you've designated a server to receive all the crash files, you should install the tool and clarify what files will store all the crash info.

  1. Verify the kdump service is running on the client.

    Kdump is automatically installed on CentOS 7 and 8. Issue the systemctl status kdump command to check the status.

    The following output should appear:

    Kdump service installation status
  2. To install the kernel-debuginfo package on both the client and server, edit /etc/yum.repos.d/CentOS-Debuginfo.repo with a text editor. Set it to enabled=1.

    The kernel-debuginfo package lets you inspect the crash dumps with the crash tool. Install the package with yum install -y kernel-debuginfo. Your interface should look like the following:

    Kernel-debug.info installation
  3. Then, modify kdump's default file upload location. To do this, edit /etc/kdump.conf, and change the default path from /var/crash to /crash-dump.

    Crash-dump file upload location

Crash the Linux kernel

Now comes the fun part: You must crash the kernel on the client.

  1. Enter both echo commands below on the client:
    echo 1 > /proc/sys/kernel/sysrq
    echo c > /proc/sysrq-trigger
    echo 1 > /proc/sys/kernel/sysrq allows for all functions of SysRq. These prompts let you send low-level commands to the Linux kernel.

    echo c > /proc/sysrq-trigger sends a sysrq command to trigger a crash.

    Because you trigger the kernel panic with echo commands, kdump should send the dump files to the NFS share.

    Connect back to the NFS server, and you can conduct a postmortem to find out what happened to the client.
  2. Check to see if the dump files upload to the NFS server.

    Within the /crash-dump directory, you should see a new directory when using the ls -lh command.

    Locate crash-dump files on NFS server
    This new directory -- 192.168.99.71-2020-04-14-12:20:47 -- originated from the client and was created during the time of the crash.

    Inside of the directory, locate the following two files: vmcore and vmcore-dmesg.text. Vmcore-dmesg.txt is the log file for dmesg during the time of the crash in plain text. Vmcore helps you use the crash utility to further investigate what processes and files .

    Here is how you locate these files:

    Find vmcore files
  3. Inspect vmcore-dmesg.txt.

    To quickly view the contents of vmcore-dmesg.txt, open the file in a text editor or grep for the word crash with the cat vmcore-dmesg.txt | grep -i crash command.

    View vmcore-dmesg.text contents
    As you can see, SysRq triggered a crash when you issued the echo commands.

Use the crash tool

The crash tool helps you analyze a crash dump file. It comes preinstalled on CentOS 8. There are also commands to run and figure out what processes and files were live at the time of the crash.

  1. On the NFS server, initialize the crash utility, and supply the following vmcore file: crash /nfs-share/vmcore /usr/lib/debug/lib/modules/4.18.0-147.5.1.el8_1.x86_64/vmlinux.

    This changes the shell prompt to the crash prompt.

    Change shell prompt to crash
  2. Then, use ps to check what processes ran.

    Check active processes with ps command
  3. The vm command shows anything loaded in the virtual memory at the time of the crash.

    Use vm command
  4. You can then use files to further examine what files were open at the time of the crash.

    View active files with files command
  5. Run the log command to see any logs again; this prompt shows the same contents as vmcore-dmesg.txt.

    Run log command

With the information from the ps, vm, files and log commands, you can figure out what caused the Linux kernel panic with the kdump crash dump tool. The NFS server lets you upload crash dump files to a server independent of the affected server. Both vmcore-dmesg.txt and vmcore files can provide a glimpse of what was going on in the system when the crash occurred.

Dig Deeper on Linux servers

Join the conversation

2 comments

Send me notifications when other members comment.

Please create a username to comment.

this is polywld7@gmail.com Paul Diaz middle name Leon...yeah what happens if i want my files back? Kernel speaking"....>>>>[]c\\//w\//m\//k\//a[]      (c)`' ;')
Cancel
Internet & Networks Google microsystems and sum......
@},--|---;------\----';'--
Cancel

  • Administrators of Office 365 or Azure platforms can benefit from using newer Microsoft software technologies to handle many jobs,...

  • Microsoft's hosting service for DNS domains is an option for organizations with a heavy Azure investment that want the benefits ...

  • Compare the features and functionality of five prominent patch management tools for Microsoft and third-party applications to ...

  • VDI has specific hardware needs that servers hosting other virtualized workloads may not meet. Learn how to gauge VDI hardware ...

  • This year's VMworld conference runs virtually from Sep. 29 to Oct. 1. Read the latest news and announcements about and from the ...

  • Admins looking for an alternative tool for emulating hardware should consider QEMU, which supports x86, PowerPC, ARM and SPARC ...

  • Get to know AWS cloud networking services for load balancing, traffic routing, content delivery and more with this overview.

  • Companies that take a careful approach to enterprise cloud adoption aligned with business objectives can help optimize spending ...

  • When it comes to SaaS spending, IT must ensure that everyone from users to management is on the same page. Use these strategies ...

Close