1.1.1NBU备份之Oracle问题故障的排查
1. Create debug folders under the following directories on database client:
阿尔山网站建设公司成都创新互联,阿尔山网站设计制作,有大型网站制作公司丰富经验。已为阿尔山1000多家提供企业网站建设服务。企业网站搭建\成都外贸网站制作要多少钱,请找那个售后服务好的阿尔山做网站的公司定做!
All log directories should be created under /usr/openv/netbackup/logs with 777 permissions.
/usr/openv/netbackup/logs/bphdb
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/dbclient
2. Attempt the backup or restore operation
3. Gather the appropriate debug information from the backup client
a. The content of all the error logs at /usr/openv/netbackup/logs
b. The backup script and the output of the backup script with exact error messages
c. /usr/openv/netbackup/bp.conf and bp.conf at home directory of oracle OS user if exists.
d. The oracle database version information
sqlplus “ / as sysdba”
SQL> select * from v$version;
SQL> select * from v$instance;
e. In case of oracle report critical errors (e.g. ORA-0600) or backup hang for a long time without response, we also need to collect the oracle alert log at $ORACLE_BASE/admin/
常用的命令
///////////////////////////////////////////////
Windows OS
调整master server的 log level,方法如下:
打开NBU Console界面, Host Properties > Master Servers, 右键master server ,点击Properties,选择 Logging, 勾选Enable robust logging ,设置Global Logging Level为 5。
创建log 目录
Install_path\netbackup\logs\bptm
Install_path\netbackup\logs\bpbrm
Install_path\netbackup\logs\vnetd
Install_path\netbackup\logs\bpcd
Install_path\netbackup\logs\bprd
Install_path\netbackup\logs\bpdbm
========
调整media server的 log level,方法如下:
打开NBU Console界面, Host Properties > Media Servers, 右键相应的media server ,点击Properties,选择 Logging, 勾选Enable robust logging ,设置Global Logging Level为 5。
创建log 目录
Install_path\netbackup\logs\bptm
Install_path\netbackup\logs\bpbrm
Install_path\netbackup\logs\vnetd
Install_path\netbackup\logs\bpcd
========
调整client server的 log level,方法如下:
在client server上,点击“开始 >所有程序>Symantec NetBackup>Backup Archive & Restore“ ,
在BAR console界面, File > Netbackup Client Properties > Troubleshooting, 设置General level为2, Verbose level为5 。
创建log 目录
Install_path\netbackup\logs\bpbkar
Install_path\netbackup\logs\bpfis
Install_path\netbackup\logs\vnetd
Install_path\netbackup\logs\bpcd
Install_path\netbackup\logs\bphdb
Install_path\netbackup\logs\dbclient
Install_path\netbackup\logs\bpdb2
Install_path\netbackup\logs\backint
///////////////////////////////////////////////
UNIX/Linux OS
调整master server的 log level,方法如下:
添加VERBOSE = 5 到 /usr/openv/netbackup/bp.conf 文件。
创建log 目录
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/vnetd
/usr/openv/netbackup/logs/bprd
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/bpdbm
需要重启 NetBackup的服务,
/usr/openv/netbackup/bin/goodies/netbackup stop
/opt/VRTSpbx/bin/vxpbx_exchanged stop
/usr/openv/netbackup/bin/bpps -x (除了pbx_exchange进程之外 ,没有其他 NB进程和 MM进程 )
/opt/VRTSpbx/bin/vxpbx_exchanged start
/usr/openv/netbackup/bin/goodies/netbackup start
========
调整media server的 log level,方法如下:
添加VERBOSE = 5 到 /usr/openv/netbackup/bp.conf 文件。
创建log 目录
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/vnetd
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
需要重启 NetBackup的服务,
/usr/openv/netbackup/bin/goodies/netbackup stop
/opt/VRTSpbx/bin/vxpbx_exchanged stop
/usr/openv/netbackup/bin/bpps -x (除了pbx_exchange进程之外 ,没有其他 NB进程和 MM进程 )
/opt/VRTSpbx/bin/vxpbx_exchanged start
/usr/openv/netbackup/bin/goodies/netbackup start
========
调整client server的 log level,方法如下:
添加VERBOSE = 5 到 /usr/openv/netbackup/bp.conf 文件。
创建log 目录
/usr/openv/netbackup/logs/bpbkar
/usr/openv/netbackup/logs/bpfis
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bphdb
/usr/openv/netbackup/logs/dbclient
/usr/openv/netbackup/logs/bpdb2
/usr/openv/netbackup/logs/backint
/usr/openv/netbackup/logs/sybackup
///////////////////////////////////////////////
socket connection failed problem
请按照以下步骤测试 NetBackup通讯的端口是否正常,
在Master Server上,
telnet client_name 13724
telnet client_name 13782
telnet media_server_name 13724
telnet media_server_name 13782
telnet media_server_name 1556
在Media Server上,
telnet client_name 13724
telnet client_name 13782
telnet master_server_name 13724
telnet master_server_name 13782
telnet master_server_name 1556
在Client Server上,
telnet master_server_name 13724
telnet master_server_name 13782
telnet master_server_name 13720
telnet media_server_name 13724
telnet media_server_name 13782
telnet master_server_name 1556
如果端口 telnet不上,请检查:
1.hosts文件中添加对端server的 hostname和ip 解析是否正确。
ping server_hostname
ping server_ip_address
2.到对端的server 上,检查telnet的端口是否 listening。
netstat -na
telnet localhost port_number
3.如果telnet localhost port_number 成功,请检查网络或 OS的防火墙是否将端口 disable
///////////////////////////////////////////////
unified logs,
====windows OS=====
调整相关 unified log level,
Install_path\netbackup\bin\vxlogcfg –a –p 51216 –o 111 –s DebugLevel=6 –s DiagnosticLevel=6
Install_path\netbackup\bin\vxlogcfg –a –p 51216 –o 116 –s DebugLevel=6 –s DiagnosticLevel=6
Install_path\netbackup\bin\vxlogcfg –a –p 51216 –o 117 –s DebugLevel=6 –s DiagnosticLevel=6
Install_path\netbackup\bin\vxlogcfg –a –p 51216 –o 118 –s DebugLevel=6 –s DiagnosticLevel=6
搜集log,
Install_path\netbackup\bin\vxlogview –p 51216 –o 111 –t 00:10:00 –d all > c:\111.txt
====UNIX/Linux OS=====
调整相关 unified log level,
/usr/openv/netbackup/bin/vxlogcfg –a –p 51216 –o 111 –s DebugLevel=6 –s DiagnosticLevel=6
搜集log,
/usr/openv/netbackup/bin/vxlogview –p 51216 –o 111 –t 00:10:00 –d all > /tmp/111.txt
//////////////////////////////////////////////
BMR bmrsaveconfig
请在这台 client上,调整bmrsavecfg 的log level
Install_path\netbackup\bin\vxlogcfg –a –p 51216 –o 121 –s DebugLevel=6 –s DiagnosticLevel=6
运行命令,如果命令报错,将报错信息也发给我。
Install_path\netbackup\bin\bmrsavecfg –infoonly
上面的命令返回提示符后,立刻运行命令搜集 log
Install_path\netbackup\bin\vxlogview –p 51216 –o 121 –t 00:10:00 –d all > c:\121.txt
搜集该主机上以下文件,
Install_path\netbackup\BareMetal\client\data\bmrcli.xml & bundle.dat
将log级别降低,
Install_path\netbackup\bin\vxlogcfg –a –p 51216 –o 121 –s DebugLevel=1 –s DiagnosticLevel=1
=======================
/usr/openv/volmgr/bin/tpconfig -emm_dev_list > /tmp/emmdev_1.txt
/usr/openv/volmgr/bin/vmoprcmd > /tmp/vmoprcmd_1.txt
/usr/openv/netbackup/bin/admincmd/bppllist -allpolicies -L > /tmp/pol.txt
/usr/openv/netbackup/bin/admincmd/bpstulist -L > /tmp/stu.txt
/usr/openv/netbackup/bin/admincmd/bperror -U > /tmp/bperror.txt
/usr/openv/netbackup/bin/admincmd/bpdbjobs > /tmp/bpdbjobs.txt
/usr/openv/netbackup/bin/goodies/available_media > /tmp/am.txt
/usr/openv/netbackup/bin/admincmd/nbemmcmd -listmedia -allrecords > /tmp/emm_media.txt
/usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts -verbose > /tmp/nbemmcmd.txt
调整nbemm,nbrb,nbjm的 log level
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 117 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 118 -s DebugLevel=6 -s DiagnosticLevel=6
同时发起 4个作业,重现两个运行 &两个排队的现象(最好这个现象可以持续 3分钟以上)
在这个现象持续的过程中,
/usr/openv/volmgr/bin/tpconfig -emm_dev_list > /tmp/emmdev_2.txt
/usr/openv/volmgr/bin/vmoprcmd > /tmp/vmoprcmd_2.txt
搜集log
/usr/openv/netbackup/bin/vxlogview -p 51216 -o 111 -t 00:10:00 -d all > /tmp/111.txt
/usr/openv/netbackup/bin/vxlogview -p 51216 -o 117 -t 00:10:00 -d all > /tmp/117.txt
/usr/openv/netbackup/bin/vxlogview -p 51216 -o 118 -t 00:10:00 -d all > /tmp/118.txt
将以上 /tmp/下的log 和命令输出文件发给我。
将后两个排队作业的 detailed status信息复制粘贴到txt文本文件发给我。
并且告诉我您同时发起的 policy的名字。
///////////////////
如果我遇到 catalog备份失败的问题,我会按照以下流程操作
1. 向用户确认,之前 catalog备份是否都成功。如果成功,需要检查这次失败的原因,或者重启一下 NBU,备份 catalog,看结果;如果新配置的,需要检查 catalog policy的配置。
2. 确认这次备份 catalog起了几个作业,应该是4个 job。让用户把 catalog失败job 的detailed status发给我们。
3. 根据status信息初步判断问题所在,如这个 case中是811
先检查一下
Install_path\netbackup\bin\admincmd\nbrbutil -dump > c:\nbrbdump.txt
Install_path\netbackup\bin\admincmd\bppllist -allpolicies -L > c:\pol.txt
Install_path\netbackup\bin\admincmd\bpstulist -L > c:\stu.txt
Install_path\netbackup\bin\admincmd\bperror -U > c:\bperror.txt
Install_path\netbackup\bin\admincmd\nbemmcmd -listhosts -verbose > c:\nbemmcmd.txt
Install_path\netbackup\bin\admincmd\bpminlicense -list_keys -verbose > c:\nbulic.txt
Install_path\netbackup\bin\admincmd\bpdbjobs > c:\job.txt
Install_path\volmgr\bin\vmglob -listall -b > c:\vmglob.txt
Install_path\volmgr\bin\tpconfig -d > c:\tpconfig.txt
Install_path\volmgr\bin\vmoprcmd > c:\vmoprcmd.txt
Install_path\volmgr\bin\tpclean -L > c:\tpclean.txt
搜集相关 log
打开NBU Console界面, Host Properties > Master Servers, 右键master server ,点击Properties,选择 Logging, 勾选Enable robust logging ,设置Global Logging Level为 5。
创建log 目录
Install_path\netbackup\logs\bptm
Install_path\netbackup\logs\bpbrm
重启NBU的服务,重新发起 NBU的 catalog备份,如果失败,将bptm和 bpbrm下的log 发给我们。
仍然失败, bptm和bpbrm 还是有811报错
需要搜集 unified log
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
调整nbemm,nbrb,nbjm的 log level
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 117 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 118 -s DebugLevel=6 -s DiagnosticLevel=6
重新发起备份,失败后
搜集log
/usr/openv/netbackup/bin/vxlogview -p 51216 -o 111 -t 00:10:00 -d all > /tmp/111.txt
/usr/openv/netbackup/bin/vxlogview -p 51216 -o 117 -t 00:10:00 -d all > /tmp/117.txt
/usr/openv/netbackup/bin/vxlogview -p 51216 -o 118 -t 00:10:00 -d all > /tmp/118.txt
/usr/openv/netbackup/bin/admincmd/nbemmcmd -listhost -verbose > /tmp/nbemmcmd.txt
/usr/openv/netbackup/bin/admincmd/bperror -U > /tmp/bperror.txt
/usr/openv/netbackup/bin/admincmd/bppllist -allpolicies -L > /tmp/pol.txt
/usr/openv/netbackup/bin/admincmd/bpstulist -L > /tmp/stu.txt
/usr/openv/netbackup/bin/bpps -x > /tmp/bpps.txt
/usr/openv/volmgr/bin/vmoprcmd -d > /tmp/vmoprcmd_me.txt
/usr/openv/volmgr/bin/tpautoconf -t > /tmp/tpautoconf.txt
/usr/openv/volmgr/bin/tpconfig -d > /tmp/tpconfig.txt
/usr/openv/volmgr/bin/scan > /tmp/scan.txt
/usr/openv/volmgr/bin/vmglob -listall -b > /tmp/vmglob.txt
/usr/openv/netbackup/bin/admincmd/nbrbutil -dump > /tmp/nbrb.txt
/usr/openv/volmgr/bin/vmoprcmd > /tmp/vmoprcmd.txt
/usr/openv/netbackup/bin/admincmd/nbemmcmd -addhost -machinename media_server_name -machinetype media -netbackupversion 7.1 -operatingsystem hpux
/usr/openv/netbackup/bin/admincmd/nbemmcmd -deletehost -machinename media_server_name -machinetype media
nbemmcmd -deletealldevices -machinename media_server_name -machinetype media
////////////////////////
将清洗带删除掉,重新按照以下配置。
1.删除清洗带
2.重新做robot inventory ,设置清洗带的类型,放入 NONE pool,清洗带类型和磁带机的类型一样,如磁带机是 hcart2的,那清洗带是1/2’’ cleaning tape 2 类型
3.双击清洗带,设置清洗次数, Number of cleanings remaining > new account ,输入一个数字,如30
4.检查清洗带剩余清洗次数 ,使用命令 install_path\volmgr\bin\vmquery -m media_id_cleaning_tape 检查输出想的 cleanings left 参数
///////////////////////
nbemmcmd -machinealias -getaliases -machinename server_name -machinetype master
//////////////////////////// emm startup failed//////////
1.请确认NBU master server 所在的磁盘的剩余空间。 NBU需要剩余空间为磁盘总空间的 10%,如 C盘 40GB size, 需要有4GB 以上的剩余空间,否则 NBU工作会异常。
2.请运行以下命令,并且将执行的命令和输出信息发给我。
Install_path\netbackup\bin\nbdb_ping > c:\nbdb_ping.txt
Install_path\netbackup\bin\bpps > c:\bpps.txt
Install_path\netbackup\bin\ipconfig /all > c:\ip.txt
Install_path\netbackup\bin\admincmd\bpgetconfig > c:\nbuconfig.txt
Install_path\netbackup\bin\admincmd\bpminlicense -list_keys -verbose > c:\nbulic.txt
Install_path\netbackupDB\data\dir > c:\db_dir.txt
Install_path\netbackupDB\log\dir > c:\log_dir.txt
3.收集以下文件发给我。
C:\windows\system32\drivers\etc\hosts
Install_path\netbackupDB\log\server.log
///////////////////////////////re-configure the devices ////////////////////////////////
1.将当前其他的备份作业取消掉,可以执行以下命令 ,
/usr/openv/netbackup/bin/admincmd/nbrbutil -resetAll
/usr/openv/netbackup/bin/admincmd/nbrbutil -dump (通过这条命令确认没有资源信息的输出)
2.删除master server 上的磁带库设备,
/usr/openv/netbackup/bin/admincmd/nbemmcmd -deletealldevices -machinename SZBK52SVC -machinetype media
通过命令检查确认,没有设备输出。
/usr/openv/volmgr/bin/tpconfig -d
3.停NBU 的服务
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/netbackup/bin/bpps -x (确认除了pbx_exchange进程,没有 NB和 MM的进程了)
/usr/openv/netbackup/bin/goodies/netbackup start
/usr/openv/netbackup/bin/bpps -x > /tmp/bpps_restart.txt
4.配置磁带库设备
/usr/openv/volmgr/bin/tpautoconf -t 检查磁带机,应该能 list出8 个磁带库的磁带机
/usr/openv/volmgr/bin/tpautoconf -r 检查机械手
/usr/openv/volmgr/bin/tpautoconf -a 将发现的设备配置到 NBU中
5.重启NBU 的服务
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/netbackup/bin/bpps -x (确认除了pbx_exchange进程,没有 NB和 MM的进程了)
/usr/openv/netbackup/bin/goodies/netbackup start
6.检查磁带机的状态
/usr/openv/volmgr/bin/tpconfig -d > /tmp/tpconfig.txt
/usr/openv/volmgr/bin/vmoprcmd > /tmp/vmoprcmd_1.txt
/usr/openv/volmgr/bin/vmoprcmd -d > /tmp/vmoprcmd_2.txt
////////////////////////////////////////////////////////
当前题目:1.1.1NBU备份之Oracle问题故障的排查
网站网址:http://scyanting.com/article/pepdio.html