1 System

1.1 Hardware

Show hardware details

lshw

show bios hardware details

dmidecode

1.2 Disk/Filesystem

Show block uage of mounted disks

df -hP

Show inode uage of mounted disks

df -hiP

show how blockdevices are configured in system

lsblk

get 10 biggest files in current dir

du -sm * .[^\.]* | sort -n | tail

use ncdu to find large files

dnf -y install epel-release
dnf config-manager --enable epel
dnf -y install ncdu
ncdu -x /  #replace / with needed mountpoint

1.3 Memory

Show memory statistics

free -th

Top provides memory information as well

top

1.4 CPU

show system load (1m, 5m, 15m)

detailed realtime statictics of procs, memory, swap, io, system, cpu

vmstat 1

1.5 Processes

tui process manager

top

show processes with tree

ps -f fax

detailed process information

ps  -faxo ruser,ppid,pid,rss,vsz,pcpu,tty,args

1.6 Network

show ip config

ip a

show wan ip

curl ifconfig.me

confgure network settings

nmtui

activate changed network settings

nmcli connection reload
nmcli device reapply <nic name>
# or just reboot the host

1.6.1 SUDO: User als admin erstellen

useradd admin
passwd admin
id admin
usermod -a -G wheel admin
id admin
su - admin
# jetzt bist du als admin eingeloggt
sudo service chronyd restart
# das geht nur als root user (sudo rechte erteilt)

2 Services

list all services

systemctl list-units --type service

List failed services on boot

systemctl --failed

show all running services

systemctl --all --state=running

show detailed service status

systemctl status <svc-name>

most common service handling

 systemctl [start|stop|restart|status|enable|disable|mask] <svc-name>

check if service is enabled to autostart after reboot

systemctl is-enabled <svc-name>

3 SELinux

show SELinux status

getenforce

disable SELinux (not persistent)

setenforce 0

change SELinux mode (reboot needed)

vi /etc/sysconfig/selinux

show selinux alerts and recomendations

sealert -a /var/log/audit/audit.log

4 Firewalling

show firewalling status

systemctl list-units --runtime | egrep 'firewall|tables'

show active iptables rules

iptables-save

show loaded nftables rules

nft list ruleset

4.1 disable firewalling for testing

check which kind of firewall is running on the system

systemctl list-units --runtime | egrep 'firewall|tables'

temporary disable ALL firewall rules (will reaper after reboot)

systemctl stop firewalld nftables iptables

test now if it works

restart all services that where running before testing (or just reboot)

systemctl start <svc-name>

5 Scheduling

5.1 cron (user/system)

cron overview

tail -n +1 /etc/crontab /etc/cron.*/* /var/spool/cron/*

edit crontab of user

crontab -e -u root

show crontab of user

crontab -l -u bob

6 at (one time)

create job

at 22:00 30.12.13

list jobs

atq

remove job

atrm <id>

7 systemd (user/system)

list timers

systemctl list-units --type timer

show configuration of timer

systemctl show <name>

get timer status

systemctl status <name>

8 Events

8.1 Show last boots

journalctl --list-boots
last reboot

8.2 Journal

Show log since last boot

journalctl -b

Kernel messages (like dmesg)

journalctl -k

Show latest log and wait for changes

journalctl -f

Reverse output (newest first)

journalctl -r

Show only errors and worse

journalctl -b -p err

Filter on time (example)

journalctl --since=2014-06-00 --until="2014-06-07 12:00:00"

Since yesterday

journalctl --since=yesterday

Show only log of SERVICE

journalctl -u SERVICE

Match executable, e.g. dhclient

journalctl /usr/sbin/dhclient

Match device node, e.g. /dev/sda

journalctl /dev/sda

8.3 Log Files

tail all currently open text files

lsof -F | sed '/^n\//!d;s/^n//;s/ .*//' | sort -u | xargs file | grep ' text$' | cut -d: -f1 | xargs tail -fn0

tail all files under /var/log

tail -f -n0 /var/log/* /var/log/*/*

9 Software

install package and deps

dnf install httpd

remove package and deps

dnf remove php-mysql

update entire system and deps

dnf upgrade

update one package without question

dnf -y update httpd

clean package cache

dnf clean all

search for name in package-name and description

dnf search gstream

search for name in package-name

dnf list '*http*'

search for package which provides this comand

dnf provides '*fortune'

list all package groups

dnf grouplist

install package group

dnf groupinstall XFCE

remove package group

dnf remove "PostgreSQL Database"

downgrade package

dnf --showduplicates list samba-common
dnf downgrade package1 package2

10 Debugging

10.1 repeat comand

watch df -hP

10.2 network bandwith monitoring

dnf -y install epel-release
dnf -y install iftop
iftop

10.3 Dstat performance monitoring

dnf -y install pcp-system-tools
dstat -af
dstat -cdngy
dstat -f -M time,cpu,net,disk,sys,swap,page,load,proc,topcpu --output $(date '+%Y.%m.%d-%H.%M')-dstat.csv
dstat -cdngymsp --lock --tcp --output $(date '+%Y.%m.%d-%H.%M')-dstat.csv

10.4 Enter debug shell (reset root password)

Reboot the machine
When grub kernel selection is showing up, press e

At the end of the line with beginning linux remove the "rhgb quiet" (on the end)
At the end of the line with beginning linux add init=/bin/bash
At the line beginning linux replace the ro with rw

Press CTRL + x to boot into this settings
Now system boots into single shell process instead of full linux env
Now you can reset the password or debug the system
hints:

su - # load full bash config

change keyboard layout

loadkeys sg #swiss german
loadkeys us #forbidden :-)

network config files

ls /etc/sysconfig/network-scripts/ifcfg-*

selinux config (disable/enable)

vi /etc/sysconfig/selinux

reset all selinux tags on next boot (relabel)

touch /.autorelabel

force filesystem check on next boot

touch /.forcefsck

force manual reboot

sync
reboot -f

10.5 strace examples (process tracing)

strace -ff -e trace=write -e write=1,2 -p SOME_PID
strace -e open ls
strace -e trace=open,read ls /home
strace -o output.txt ls 
strace -f -p 1725 -o firefox_trace.txt # f: follow process
strace -t -e open ls /home #timestamp
strace -c ls /home # statistics
strace -f -t -e trace=file systemctl restart SuSEfirewall2 2>&1 | grep open | cut -d'"' -f2

10.6 SOS Report (Log Bundle)

This collects a log bundle which can be used for later debugging and analyzing.
Start this BEFORE you try to repair or change something, as well do not forget to snapshot/backup System/Application before debugging.

dnf -y install sos
sosreport
cp -av /var/tmp/sosreport* /root/

CentOS8 CheatSheet fuer System Administratoren

Contents