VCSA 6x Troubleshooting Free
Jump to navigation
Jump to search
Contents
- 1 General
- 2 Photon OS Administration
- 3 tdnf basics
- 4 Photon OS Troubleshooting
- 5 vCenter 6.x VCSA
- 5.1 vCenter 6.x Max Value difference
- 5.2 vCSA OS status archiver
- 5.3 vCSA Backup / Restore script
- 5.4 Service Control
- 5.5 VCSA Services
- 5.6 Enable SSH
- 5.7 Modify vCSA Iptables rules
- 5.8 Change root shell
- 5.9 WinSCP transfer fails
- 5.10 replace certificate on vcsa
- 5.11 SSL/TLS debugging
- 5.12 Upgrade VCSA 6.0 to 6.5
- 5.13 VAMI comandline access
- 5.14 Appliance Shell examples
- 5.15 Install ssh public key manually
- 5.16 Troubleshooting full filesystem
- 5.17 Access/Troubleshoot PostgreSQL DB
- 5.18 vCSA troubleshooting KBs
- 5.19 vCSA Important KBs
- 5.20 VCSA KBs to review
- 5.21 vCSA KBs to discuss
- 6 More fun
1 General
1.1 General Documentation
1.2 Release Notes
1.3 VMware Links
- Latest VMware Tools
- Build numbers and versions of VMware products
- VMware Tools version number codes
- VirtuallyGettho Scripts- READ IT
1.4 LAB Links
- Virtually Ghetto - Nested Virtualization
- Virtually Ghetto - Shared VMDK
- Virtually Ghetto - Nested Esxi - Promiscuous Mode & Forged Transmits
- Shared VMDK Files for cheap SAN emulation - KB
- VMUG - 200 USD / Year - "All" VMware products
2 Photon OS Administration
2.1 Console basics
- Swtich TTY
ALT+F1 -> ... -> ALT+F12 chvt 5 # change to TTY 5 echo here I am > /dev/tty5
- change keyboard layout on console
loadkeys us loadkeys sg loadkeys --help
- redirect kernel logs to TTY
dmesg -Tw | grep -v IPfilter > /dev/tty6 & # redirect kernel logs to tty6 and put job into background disown %1 # detacht the job above and make it save to logout chvt 5 # switch vm console to tty5 (so humans can inspect the logs)
2.2 VIM basics
<Esc> command mode V visual mode i insert mode : command line mode :help<Enter> find help :r! date -R insert date command output qa record into register a q quit recorder @a replay recorder with register a :edit text.txt edit next file :split /new open new file in splitted screen :wnext save current file and jump into next one % s/from/to/g replace string in whole file 10g<Enter> jump to line 10 /string search string :wq save and exit :q! do NOT save and exit
2.3 Comandline task management
<CTRL>+<Z> #detach job (paused) bg #background job (running) jobs #show jobs fg #catch job (fg) kill -9 %1 #kill job1 disown %1 #detach job1 from tree (safe for logout) nohup %1 #detach job1 from tree (safe for logout)
2.4 RPM basics
- list all packages
rpm -qa
- list last installed packages
rpm -qa --last | head
- find rpm which is owning a file
rpm -qf /etc/dnsmasq.conf
- check rpm for changed files
rpm -V dnsmasq
- show all files of rpm
rpm -ql dnsmasq
- show package info and config files, found by file
rpm -qif --configfiles /opt/vmware/etc/init.d/vami-lighttp
- show all vmware config files, owned by rpms
rpm -qi --configfiles lighttpd
- show all scripts, owned by rpms
rpm -qi --scripts lighttpd
3 tdnf basics
- look here tdnf_package_handling
4 Photon OS Troubleshooting
4.1 Default troubleshooting tools
- tcpdump -C 1024 -w /tmp/vcsa_dump.dat -i eth0 -s 0 host 10.77.89.34
- netcat
- strace -e open systemctl restart vami-lighttp
- lsof -i -P -n | grep LISTEN
- netstat -taupen
- top
- lsblk
- blkid
- dmesg -Tw
4.2 Logins/Reboots
- List last Logins / Reboots
last
4.3 Hardware
- kernel log
dmesg -T
- HW info
dmidecode cat /proc/cpuinfo
- disk information
lsblk blkid df -PTha df -PThai cat /proc/partitions
4.4 Services
- get all systemd objects
systemctl list-unit-files
- get running services
systemctl --all
- get failed services
systemctl --all --state=failed
- useful cmds
systemctl | grep ' vm' systemctl list-dependencies SVC_NAME
- systemd dirs
- /etc/systemd/system/
- /usr/lib/systemd/system/
4.5 Logs
- show recent changed log files
find /var/log -follow -type f -mmin -1
tail -f /storage/log/vmware/applmgmt/monsvc.log | egrep --color=always -i '$|error|crit|warn'
lsof | egrep 'log$|out$' | awk '{print $10}' | sort -u | xargs tail -f | egrep --color=always -i '$|error|crit|warn'
lsof /var/log /storage/log | awk '{print $9}' | sort -u | xargs tail -f | tee /tmp/all.log
journalctl -af
4.6 User / Group
- get local user/group details
getent passwd getent group id $USER chage -l $USER
- files
/etc/passwd # user config /etc/shadow # user# passwd /etc/group # group config /etc/gshadow # group password
- unlock user account
- Minimum Password Age to 0 (m)
- Maximum Password Age to 99999 (M)
- Password Inactive Lock after X days to -1 (I)
- Account Expiration Date to -1 (E)
chage -I -1 -m 0 -M 99999 -E -1 username
4.7 Process / Performance
- dstat live performance
dstat -cdngysl --nocolor dstat -cdngymsp --lock --tcp dstat -f -M time,cpu,net,disk,sys,swap,page,load,proc,topcpu --output $(date '+%Y.%m.%d-%H.%M')-dstat.csv # convert csv2html: https://github.com/joe-speedboat/scripts/blob/master/dstat-csv2gnuplot.sh # with vcsa 6.7, you need python2 as well: tdnf install python2 # not supported by VMw
- Process view
top vimtop ps faxu | less
- Process trace
strace -f -p SOME_PID 2>&1 | grep string strace -e trace=open,read -p SOME_PID strace -c find /usr/local/ -type f strace -c ping -c3 -w3 8.8.8.8 strace -e trace=open,read,stat,file -p SOME_PID
4.8 Networking
- sockets and ports and connection status
ss netstat -taupen lsof -i -P -n
- tcpdump network analysis
tcpdump -C 100 -w sw1247_dump.dat -i eth0 -s 1500 host 10.77.89.34 tcpdump host client.example.com and port 80 tcpdump -x -s0 ip host vm05 and host vm06 and \( port 80 or port 8080 \) tcpdump ip and not net 127.0.0.0 tcpdump not port 22
- outgoing connectivity (tcp port)
nc -zv 1.2.3.4 443
4.9 Backup / Restore
- when ever possible make snapshot of vcsa before troubleshooting
- take care of permissions, use always cp -a
4.10 Reset Root Password
- vcsa 6.5 trying to automate the backup - authentication failure
- VMware RootPW recovery KB
- Take Snapshot of the virtuell vCenter Appliance
- Connect to ESXi Server running the vCenter Appliance
- Connect to the vCenter Appliance Console
- Reboot the vCenter Appliance
- When the PhotonOS Screen is showing, press e
- At the end of the line with beginning linux add rw init=/bin/bash
- Press CTRL + x
- Type passwd
- Enter new root Password
- umount /
- reboot -f
- hint: you an mount a ro filesystem into rw state with: mount -o remount,rw /
4.11 Unlock user after entering emergency shell
- show user account state
chage -l root
- unlock user
usermod -U root
- diable password expiration
chage -M 9999 root
5 vCenter 6.x VCSA
5.1 vCenter 6.x Max Value difference
| Type | vCenter 6.0 | vCenter 6.5 | vCenter 6.7 |
|---|---|---|---|
| OS | SUSE Linux Enterprise 11 | Photon OS 1.0 | Photon OS 1.0 |
| MGMT | Web Client / FAT Client | Web Client / HTML5 Client | Web Client / HTML5 Client |
| max CPU/Host | 480 | 576 | 768 |
| max vCPUs/VM | 128 | 128 | 128 |
| max VMs/Host | 1024 | 1024 | 1024 |
| max vCPUs/Core | 32 | 32 | 32 |
| max vCPUs/Host | 4096 | 4096 | 4096 |
| max Mem/Host | 12TB | 12TB | 16TB |
| max Hosts/Cluster | 64 | 64 | 64 |
| max VMs/Cluster | 8000 | 8000 | 4000 |
| vMEM/VM | 4TB | 6TB | 6TB |
| REST API | No | Yes | Yes |
5.2 vCSA OS status archiver
- https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_status.sh
- https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_status.sh
5.3 vCSA Backup / Restore script
- YOU NEED TO KEEP INSTALL MEDIA AS WELL
- Backup
- https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_backup_scp.sh
- https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_backup_scp.sh
- From within VCSA: http://www.vmwarearena.com/schedule-vcsa-6-7-backup/
- Restore
- https://github.com/joe-speedboat/scripts/blob/master/vcsa_65_restore_scp.sh
- https://github.com/joe-speedboat/scripts/blob/master/vcsa_67_restore_scp.sh
- Debug Log: /storage/log/vmware/applmgmt/backupRestoreAPI.log
WARNING: PNID and system.name must be equal to run restore
[root@deployment vcenter]# grep vcenter1 /media/vmware/vcenter/json/vcenter1.json
"name": "vcenter1"
"system.name": "vcenter1.lab.local"
backup-host:~/vcenter1.lab.local/2018-04-24-14-07-35 # grep vcenter1.lab.local backup-metadata.json
"PNID": "vcenter1.lab.local",
5.4 Service Control
- topping, starting, or restarting VMware vCenter Server Appliance 6.x services KB2109887
- Stopping, starting, or restarting services in vCenter Server Appliance 6.5 KB2147152
- list services
service-control --list
- check status of services
service-control --list | cut -d' ' -f1 | while read svc do systemctl status $svc echo "--------------------------------" done
| Photon OS Commands | SLES Commands |
|---|---|
| systemctl start <service> | service <service> start |
| systemctl stop <service> | service <service> stop |
| systemctl restart <service> | service <service> restart |
| systemctl reload <service> | service <service> reload |
| systemctl condrestart <service> | service <service> condrestart |
| systemctl status <service> | service <service> status |
| systemctl enable <service> | chkconfig <service> on |
| systemctl disable <service> | chkconfig <service> off |
| systemctl daemon-reload | chkconfig <service> --add |
| systemctl list-units --type | service --all chkconfig --list <service name> |
| systemctl is-enabled <service> | chkconfig --list vmware-tools-services |
5.5 VCSA Services
| Service Name | Description |
|---|---|
| applmgmt | VMware Appliance Management Service |
| vmware-cis-license | VMware License Service |
| vmware-cm | VMware Component Manager |
| vmware-eam | VMware ESX Agent Manager |
| vmware-sts-idmd | VMware Identity Management Service |
| vmware-invsvc | VMware Inventory Service |
| vmware-mbcs | VMware Message Bus Configuration Service |
| vmware-netdumper | VMware vSphere ESXi Dump Collector |
| vmware-perfcharts | VMware Performance Charts |
| vmware-rbd-watchdog | VMware vSphere Auto Deploy Waiter |
| vmware-rhttpproxy | VMware HTTP Reverse Proxy |
| vmware-sca | VMware Service Control Agent |
| vmware-sps | VMware vSphere Profile-Driven Storage Service |
| vmware-stsd | VMware Security Token Service |
| vmware-syslog | VMware Common Logging Service |
| vmware-syslog-health | VMware Syslog Health Service |
| vmware-vapi-endpoint | VMware vAPI Endpoint |
| vmware-vdcs | VMware Content Library Service |
| vmafdd | VMware Authentication Framework |
| vmcad | VMware Certificate Service |
| vmdird | VMware Directory Service |
| vmware-vpostgres | VMware Postgres |
| vmware-vpx-workflow | VMware vCenter Workflow Manager |
| vmware-vpxd | VMware vCenter Server |
| vmware-vsm | VMware vService Manager |
| vsphere-client | vSphere Web Client |
| vmware-vws | VMware System and Hardware Health Manager |
| vmware-vsan-health | VMware vSAN Health Service |
5.6 Enable SSH
- WebUI
- https://vcenter/vsphere-client
- Administration > System Configuration > Nodes > vCenter > Manage > Settings > Access
- VMI
- https://vcenter:5480
- Access > SSH
5.7 Modify vCSA Iptables rules
- this is not perisistent across reboots!
iptables-save > iptables.rules # save current rules to file iptables-save | grep 443 # find rules for https iptables -D port_filter -p tcp -m tcp --dport 443 -j ACCEPT # delete https input rule (replace -A with -D) ip a # find your ip iptables -A port_filter -p tcp -m tcp -s 192.168.3.220 --dport 443 -j ACCEPT # inject custom rule with your ip # now only your ip can access https interface (be aware of APIs which may also fail) iptables-restore < iptables.rules # restore old rules when work done
5.8 Change root shell
5.9 WinSCP transfer fails
NOTE: with bash shell, you wont hit this issue
- Log in vCSA VAMI UI (https://vcsa-ip:5480)
- Under Access, enable SSH Login. (PS: enabling Bash Shell is not necessary)
- Open WinSCP, select File protocol: SFTP
- Enter the vCSA hostname, port number 22, root, and root’s password
- Click Advanced
- Under Environment, SFTP, Protocol options
- Set SFTP server to shell /usr/libexec/sftp-server (vcsa 6.5)
- Set SFTP server to shell /usr/libexec/sftp-server (vcsa 6.7)
- Set SFTP server to shell /usr/lib64/ssh/sftp-server (vcsa 6.0)
5.10 replace certificate on vcsa
5.11 SSL/TLS debugging
- Get all LISTEN Ports and test for Certificate details by web request
IF=$(ip r | grep default | awk '{print $5}')
IP=$(ip addr show $IF | grep "inet " | awk '{print $2}' | cut -d'/' -f1)
lsof -i -P -n 2>/dev/null | grep LISTEN | grep -v 127.0.0.1 | awk '{print $9}' | cut -d: -f2 | sort -u | while read port
do
echo "---------- https://$IP:$port "
lsof -i -P -n 2>/dev/null | grep LISTEN | grep :$port
curl -m3 -k -vv https://$IP:$port 2>&1 | grep -A4 'Server certificate:'
done
5.12 Upgrade VCSA 6.0 to 6.5
- Best practices for upgrading to vCenter Server 6.5 2147686
- Full Patch for VMware vCenter Server Appliance 6.5.0d KB2149221
- tinkertry - Upgrade VCSA 6.0
- cloudhat.eu - Upgrade VCSA 6.5
- Note: For production, it is recommended to upgrade with update iso
5.13 VAMI comandline access
- * WARNING: you easily f... .p the VCSA with this commands
- show all the VMI cli comands
ls -1 --color=never /opt/vmware/share/vami/vami_*
- some useful read cmds
/opt/vmware/share/vami/vami_dns /opt/vmware/share/vami/vami_get_network eth0 /opt/vmware/share/vami/vami_set_proxy /opt/vmware/share/vami/vami_set_timezone
5.14 Appliance Shell examples
- this is the recommended way to change things inside vcsa
Command> help api list Command> ntp.get Command> ntp.server.set --help
- Script your config
vim vcsa_api.cmd
------
dns.domains.set --domains office.bitbull.ch
ntp.set --servers pool.ntp.org
dns.domains.list
ntp.get
------
cat vcsa_api.cmd | ssh root@vc05
Pseudo-terminal will not be allocated because stdin is not a terminal.
VMware vCenter Server Appliance 6.7.0.30000
root@vc05's password:
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"
Command> Command> Command> Domains: office.bitbull.ch
Command> Servers: pool.ntp.org
5.15 Install ssh public key manually
- Client host, create key pair
ssh-keygen cat .ssh/id_???.pub # copy rsa key to clipboard
- on target system
cd /root mkdir .ssh chmod 700 .ssh cd .ssh touch authorized_keys chmod 600 authorized_keys echo 'my-super-public-key' >> /root/.ssh/authorized_keys
- keep in mind
- it only works with bash as shell and not with appliance shell from vmware
- https://kb.vmware.com/s/article/2146205
5.16 Troubleshooting full filesystem
5.16.1 Find big files
root@vc04 [ /storage/log ]# df -Th | grep ext
/dev/sda3 ext4 11G 5.1G 5.0G 51% /
/dev/mapper/log_vg-log ext4 9.8G 9.8G 0G 100% /storage/log
/dev/mapper/updatemgr_vg-updatemgr ext4 99G 82M 94G 1% /storage/updatemgr
/dev/mapper/autodeploy_vg-autodeploy ext4 9.8G 23M 9.2G 1% /storage/autodeploy
/dev/sda1 ext4 120M 34M 78M 31% /boot
/dev/mapper/netdump_vg-netdump ext4 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/core_vg-core ext4 25G 88M 24G 1% /storage/core
/dev/mapper/imagebuilder_vg-imagebuilder ext4 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/db_vg-db ext4 9.8G 102M 9.1G 2% /storage/db
/dev/mapper/seat_vg-seat ext4 9.8G 55M 9.2G 1% /storage/seat
/dev/mapper/archive_vg-archive ext4 50G 64M 47G 1% /storage/archive
/dev/mapper/dblog_vg-dblog ext4 15G 102M 14G 1% /storage/dblog
root@vc04 [ /storage/log ]# cd /storage/log/vmware
root@vc04 [ /storage/log ]# du -mcs * | sort -n | tail
1 lost+found
1 remote
1 vcsa_67_status-week-20.log
1 vcsa_67_status-week-21.log
9905 vmware
9905 total
root@vc04 [ /storage/log/vmware ]# du -mcs * | sort -n | tail
3 vsphere-ui
4 vapi
5 procstate-20190521.gz
7 vpxd
7 vsphere-client
9 vsan-health
12 sso
41 procstate
9807 cis-license
9905 total
root@vc04 [ /storage/log/vmware/cis-license ]# du -mcs * | sort -n | tail
1 license.log.1.gz
1 license.log.2.gz
1 license-runtime.log-0.stderr
1 license-runtime.log-0.stdout
1 license-runtime.log-1.stderr
1 license-runtime.log-1.stdout
1 license-runtime.log.stderr
1 license-runtime.log.stdout
9807 total
9807 vmware-cis-license-gc.log.0.current
# File found. Is it safe to delete it?
root@vc04 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.0.current
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
vmware-ci 2095 root 3w REG 252,6 10282348544 524436 vmware-cis-license-gc.log.0.current
5.16.2 truncate big file which is in use
- Oh no, file is open if we delete, process will probably freak out
- so we truncate the file instead of deleting it
# File found. Is it safe to delete it?
root@vc04 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.0.current
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
vmware-ci 2095 root 3w REG 252,6 10282348544 524436 vmware-cis-license-gc.log.0.current
root@vc04 [ /storage/log/vmware/cis-license ]# ls -lahi vmware-cis-license-gc.log.0.current
524453 -rw-r--r-- 1 root root 9.7G May 21 11:39 vmware-cis-license-gc.log.0.current
root@vc04 [ /storage/log/vmware/cis-license ]# > vmware-cis-license-gc.log.0.current
root@vc04 [ /storage/log/vmware/cis-license ]# ls -lahi vmware-cis-license-gc.log.0.current
524453 -rw-r--r-- 1 root root 0 May 21 11:41 vmware-cis-license-gc.log.0.current
root@vc04 [ /storage/log/vmware/cis-license ]# df -hT .
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/log_vg-log ext4 9.8G 54M 9.2G 1% /storage/log
5.16.3 delete logfiles which are not in use (should not be necessary)
- Try first: manually kick log rotation (new in v6.7)
root@vc05 [ /etc/cron.d ]# ls -lah /etc/cron.d/*log*
-rw-r--r-- 1 root root 62 Mar 27 05:30 /etc/cron.d/logdiskcheck.cron
-rw-r--r-- 1 root root 97 May 20 13:21 /etc/cron.d/logrotate.cron
root@vc05 [ /etc/cron.d ]# cat /etc/cron.d/logdiskcheck.cron
*/10 * * * * root /usr/sbin/logdiskcheck.sh >/dev/null 2>&1
root@vc05 [ /etc/cron.d ]# cat /etc/cron.d/logrotate.cron
0 */1 * * * root /usr/sbin/logrotate /etc/logrotate.conf 2>&1 | /usr/bin/logger -t logrotate
root@vc05 [ /etc/cron.d ]# cat /usr/sbin/logdiskcheck.sh
log_dir="/var/log/vmware/"
max_usage_percent=90
usage_percent=$(df $log_dir | tail -1 | awk '{print $5+0}')
if [ $usage_percent -lt $max_usage_percent ]; then
exit 0
fi
/usr/bin/logger -t logdiskcheck "WARN logdisk nearly full. Attempt to cleanup."
# Force a log rotate.
/usr/sbin/logrotate -f /etc/logrotate.conf
# Delete more than a week old hprof files from component log directory.
find "$log_dir" -mindepth 2 -mtime +7 -type f -name '*.hprof' -delete
root@vc05 [ /etc/cron.d ]# sh -x /usr/sbin/logdiskcheck.sh
+ log_dir=/var/log/vmware/
+ max_usage_percent=90
++ df /var/log/vmware/
++ awk '{print $5+0}'
++ tail -1
+ usage_percent=100
+ '[' 100 -lt 90 ']'
+ /usr/bin/logger -t logdiskcheck 'WARN logdisk nearly full. Attempt to cleanup.'
+ /usr/sbin/logrotate -f /etc/logrotate.conf
error: error creating output file /var/log/vmware/vpostgres/pg_archiver.log.stderr.1: File exists
error: error creating output file /var/log/vmware/vpostgres/pg_archiver.log.stdout.1: File exists
error: error creating output file /var/log/vmware/vpostgres/postgres_cron.log.1: File exists
+ find /var/log/vmware/ -mindepth 2 -mtime +7 -type f -name '*.hprof' -delete
root@vc05 [ /etc/cron.d ]# df /storage/log
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/log_vg-log 10181944 10164068 0 100% /storage/log
This is NORMALLY not needed, VMware manually kicks logrotation every 10min if filesystem /storage/log/vmware is filled more than 90%
BUT BE AWARE: not all log files are maintained by vmware config, ours is not!!!
5.16.4 delete logfiles manually (last option)
- big file is not used by any process (lsof big-file)
- logrotate script (step above) did not cleanup the disk filling problem
root@vc05 [ /storage/log/vmware/cis-license ]# ls -lah vmware-cis-license-gc.log.1
-rw-r--r-- 1 root root 9.7G May 21 12:39 vmware-cis-license-gc.log.1
root@vc05 [ /storage/log/vmware/cis-license ]# lsof vmware-cis-license-gc.log.1 #file must not be in use
# truncating log is better then deleting it. logrotate may expect proper file numbering
root@vc05 [ /storage/log/vmware/cis-license ]# > vmware-cis-license-gc.log.1
root@vc05 [ /storage/log/vmware/cis-license ]# ls -lah vmware-cis-license-gc.log.1
-rw-r--r-- 1 root root 0 May 21 12:53 vmware-cis-license-gc.log.1
root@vc05 [ /storage/log/vmware/cis-license ]# df -hT .
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/log_vg-log ext4 9.8G 58M 9.2G 1% /storage/log
5.17 Access/Troubleshoot PostgreSQL DB
- Keep an eye on file permissions when changing configurations
- PG Access KB2147285
- Selective deletion of tasks, events, and historical performance data in vSphere 5.x and 6.x KB2110031
- fetch credentials as root user
cat /etc/vmware-vpx/embedded_db.cfg
- swith into disabled vpostgres user
su -s /bin/bash vpostgres
- connect database
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
5.17.1 log purge on vCSA 6.x
curl -s "https://kb.vmware.com/servlet/fileField?entityId=ka234000000TvzOAAS&field=Attachment_3__Body__s" > 2110031_Postgres_task_event_stat.sql curl -s https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/queryVCDBvPostgres.sh | sh /opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=7 -v EventMaxAgeInDays=7 -v StatMaxAgeInDays=7 -d VCDB -t -q -f 2110031_Postgres_task_event_stat.sql curl -s https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/queryVCDBvPostgres.sh | sh
5.17.2 purge vcsa logs by truncating (not verified by VMware)
- If db is full then regular kb will lead into error because temporary space is needed but not available
- Find an idea what to do and how others do get around the problem:
- get an idea on what to delete
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c '\dt
- prepare truncate psql statement as you need copy paste into sql prompt:
[chris@lts1 ~]$ seq 100 | while read nr; do echo "truncate table vpx_event_$nr cascade;"; done truncate table vpx_event_1 cascade; [...] truncate table vpx_event_100 cascade;
- Open SQL Prompt and paste the comand from step above
- Example db login:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
psql.bin (9.4.9 (VMware Postgres 9.4.9.0-4442725 release))
Type "help" for help.
VCDB=#
SELECT * FROM VPX_VERSION;
SELECT * FROM VPX_HOST;
\q
5.18 vCSA troubleshooting KBs
- Troubleshooting Tools for Networking on vCenter Server Appliance 6.5 KB2147148
- Finding vCenter when it's Down
5.19 vCSA Important KBs
- VMware vCenter Server 6.5 Update 1g Release Notes
- Troubleshooting Tools for Networking on vCenter Server Appliance 6.5 KB2147148
- Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.5 KB2145603
5.20 VCSA KBs to review
- Supported and deprecated topologies for VMware vSphere 6.5 KB2147672
- OVF deployment fails after upgrading to vCenter Server Appliance 6.5 U1 KB2151085
- Email alerts for alarms are not delivered on vCenter Server Appliance 6.5 KB2148396
- Manually running fsck on vCenter Server Appliance 6.5 KB2147154
- VUM option not available in Web Client after upgrade to VCSA 6.5 KB2151923
- Resetting VMware Update Manager Database on a vCenter Server Appliance 6.5 KB2147284
5.21 vCSA KBs to discuss
- vCenter Server Appliance 6.5 fails to boot with error: Failed to start Network Service KB2150986
- The vCenter Inventory Service fails to start or slow to start KB2032828
6 More fun
6.1 ESXi console
- How to back up ESXi host configuration KB2042141
- Some small functions and helpers I wrote: EsxiNotes
6.2 OpenSSL comandline
- Things I broke and/or fixed: Openssl_Notes
6.3 LAB exercises to discuss
mount | column -t
ls -lR / >std_out.log 2>err.log cat std_out.log cat err.log
lsof -i -P -n | grep LISTEN | egrep -v '::1|127.0.0.1' | awk '{print $1 " "$9}' | sort -u
find /var/log /storage/log -follow -type f -mmin -5
find /var/log /storage/log -follow -type f -mmin -5 | xargs file | grep ASCII | cut -d: -f1 | xargs tail -f
lsof | egrep 'log$|out$' | awk '{print $10}' | sort -u | xargs tail -f | egrep --color=always -i '$|error|crit|warn'
lsof /var/log /storage/log | awk '{print $9}' | sort -u | xargs tail -f | tee -a /tmp/all.log
reset
curl ifconfig.me
lsof /storage/log/vmware/vapi/endpoint/*
# log your work some-important-cmd | tee -a /root/debug.log script -a /root/my_session.script
