This chapter covers the following primary troubleshooting topics:
Troubleshooting network problems is by no means an easy task. There are numerous areas where network problems can arise. Bad wiring causes problems at the physical layer. A failed network card results in trouble at the data-link layer. Routing errors arise at the network layer. Worse yet, the network operating system or applications may cause problems at the transport, session, presentation, or application layer.
As this litany of potential woes indicates, it's important to have a good understanding of the OSI model (one of the subjects of Chapter 4, "Choosing Network Protocols") when you troubleshoot network problems. Finding a network problem often involves deciding which OSI layer is not the culprit. You need good troubleshooting tools and intimate familiarity with the networking protocols in use before you can be an effective network troubleshooter.
This chapter concentrates on solving the most common networking problems associated with Windows NT Server 4.0. Fortunately, Windows NT Server provides several tools (such as Control Panel) and utilities (such as Network Monitor) for configuring and troubleshooting network problems.
Windows NT and its predecessor, LAN Manager, rely on NetBIOS protocols-including Server Message Blocks (SMBs), named pipes, and mailslots-for all native file and print services, plus many of Windows NT Server's application services. Most NetBIOS-based protocol stacks, such as Windows NT's NetBEUI Frame (NBF), rely on broadcasting network packets to every client and server on the network.
Each NetBIOS device on the Windows NT network has a NetBIOS name, which uniquely identifies the device and contains information about the NetBIOS-related services the device provides. Broadcasting is the means by which a NetBIOS device advertises its NetBIOS name and capabilities to other network devices. Broadcasting is also how a NetBIOS device can locate another device or capability on the network.
Broadcasting is ideal for services that require real-time distribution to multiple clients, such as real-time stock-market data. Broadcasting is well suited to small networks because it reduces network response time. Broadcasting, with reduced packet overhead, is responsible for NetBEUI's excellent performance on small networks. On larger networks, broadcasting creates a substantial amount of network traffic and adds to the difficulty of identifying a network device that's causing a problem. Much of your network troubleshooting time-if you have a large network-is likely to be devoted to overcoming problems related to excessive broadcasts.
All NetBEUI traffic is broadcast-based. If you install only the NetBEUI protocol on your Windows NT server, the server responds only to broadcast requests from other NetBEUI devices. NetBEUI isn't routable; if you have two Ethernet segments separated by a router (see fig. 15.1), NetBEUI broadcasts aren't forwarded across the router. The only way to overcome this limitation is to enable bridging on the router, so that both segments appear as one physical segment. In this case, the router bridges the traffic between the two segments using MAC (Media Access Control) layer addresses. Bridging eliminates the advantages of a segmented network. As you can imagine, maintaining a bridged network in a large LAN installation quickly grows unmanageable, especially if you also must maintain routed protocols, such as TCP/IP, on the same segments.
See "NetBEUI and NetBEUI Frame," (Ch 4)
NetBEUI broadcast blocking on a routed network.
If you must troubleshoot problems with a broadcast-based protocol, most of the tools for a directed protocol, such as TCP/IP, are ineffective. This is because many conventional network analysis tools rely on analyzing the interaction of source-destination network addresses to determine the flow of communication and where a connection is failing. Broadcast-based systems simply flood packets to every network device, whether or not the device is involved in the connection.
The best way to troubleshoot problems on a broadcast network is to connect a protocol analyzer to the network, and then filter MAC addresses to determine who transmitted what to whom. Protocol analyzers are one of the subjects of the later section "Using Windows NT's Troubleshooting Tools."
The best way to troubleshoot broadcast-based network problems with a protocol analyzer is to filter out all traffic except those devices you're interested in. For example, with Microsoft's Network Monitor, you can filter on a given source and destination MAC address pair if you know those devices are having network problems.
Windows NT 4.0's IPX/SPX support is similar to that for NetBEUI, except that IPX implements the OSI network layer, whereas NetBEUI doesn't. The addition of the network layer makes IPX routable. Windows NT's NWLink stack supports the IPX protocol with the NetBIOS session layer. This means that you can run your NT network using only IPX for file, print, and application services, if you choose.
See "NWLink (IPX/SPX)," (Ch 4)
In this case, although the IPX/SPX protocol is routable, NetBIOS-over-IPX is broadcast-based. NetBIOS service announcement functions, such as browsing, aren't forwarded across subnets in a routed network. This means that devices running only NWLink and separated by a router won't normally be able to browse or connect to each other's resources. You must have a mechanism on your dedicated routers, such as that provided by Cisco Systems' IPX Helper feature, to forward broadcasts to the desired destination.
TCP/IP is a directed protocol that eliminates most of the broadcast traffic associated with NetBEUI and IPX protocols. A directed protocol usually involves point-to-point communication between two or more networked devices. Integration of NetBIOS and TCP/IP, which Microsoft calls NetBT (or, more commonly, NBT) follows two established Request For Comments (RFCs), defined by the Internet Engineering Task Force (IETF) as RFCs 1001 and 1002. From these RFCs, Microsoft built an entire suite of services to facilitate the use of TCP/IP with NetBIOS.
See "TCP/IP," (Ch 4)
The RFCs for NBT specify three TCP/IP service ports, which perform the following functions:
By using these three ports, Windows NT provides all its native services over TCP/IP. Windows NT also supports more traditional TCP/IP services, such as FTP and Telnet.
For a list of all the TCP/IP services Windows NT supports, view the %systemroot%\system32\drivers\etc\Services file. This file, like its UNIX counterpart, contains service names and port numbers for most TCP/IP services. Services is a simple text file, so you can view it with a text editor, such as Notepad.
Use of TCP/IP requires traditional NetBIOS functions such as network browsing, name lookups, and user messaging to be mapped to the network addresses used by TCP/IP. Microsoft developed the Windows Internet Naming Service (WINS) as its method for mapping IP addresses to NetBIOS machine names. The function of WINS is similar to the Domain Name Service (DNS) provided by most UNIX-based systems, but WINS provides additional NetBIOS-related services, including the capability to associate different NetBIOS functions for a given machine name to a single IP address and to register Windows NT domain names to an IP address that represents a domain controller in the specified domain.
Machines running Windows NT Server and Workstation with the TCP/IP protocol stack use WINS to register their NetBIOS names and IP addresses. WINS registration occurs dynamically when the device starts up, or statically to guarantee that a certain machine name is registered to a certain IP address. WINS also registers Windows users based on the IP address of the machine from which each user logs on. This feature is used by services that must determine the IP address for a given user. For instance, the net send command queries WINS to find the IP address for the message destined for a specified user name.
After all servers and clients are registered with WINS, subsequent NetBIOS-related operations such as browsing, messaging, authentication, and file and print services use the WINS database, located on a Windows NT server, to perform name resolution between NetBIOS names and IP addresses. Using the WINS database eliminates the need for broadcast name resolution, thus decreasing network traffic and easing troubleshooting tasks.
If you're familiar with the LMHOSTS file used by LAN Manager, you can consider WINS to be a centralized version of LMHOSTS. WINS provides centralized name resolution without having to maintain individual LMHOSTS files on each Windows computer. Windows NT automatically uses the LMHOSTS file, if needed.
The WINS database is a Jet (Access) database, WINS.MDB, and uses a Jet system database with the default name, SYSTEM.MDB, both of which are located in the \%systemroot%\system32\wins folder. Figure 15.2 shows an example view of a WINS database that includes entries for workstations, users, and domains. Assuming that you've installed WINS, follow these steps to view the WINS database:
A WINS database viewed in WINS Manager's Mappings dialog.
WINS.MDB grows in size as old records are deleted and new records are added, a characteristic of all Jet databases. The initial size of the database is about 2M. When the size of the database exceeds about 10M, stop the WINS service and run jetpack wins.mdb winspack.mdb from the \%systemroot%\system32\wins folder to compact the WINS database. (The WINSPACK.MDB file is a temporary database file to which WINS.MDB is compacted; WINSPACK.MDB replaces WINS.MDB and is deleted if the compaction is successful.) Periodically compacting the WINS database improves performance.
The hexadecimal value (indicated by the suffix h) in square brackets ([]) after each name shown in figure 15.2 indicates the type of service the entry provides. This value is called the 16th Byte of the name. Table 15.1 defines each device name type and 16th Byte value you're likely to encounter.
Table 15.1 WINS 16th Byte Values for Domain, User, and Machine Names
Device Name and 16th Byte Value |
Usage |
Domain Names |
|
<domain_name | [1Dh] The name of the Domain Master Browser. |
<domain_name | [1Eh] The name used by browsers to elect a Master Browser. |
<domain_name | [00h] The name registered by the Workstation service on the domain controller to receive browser broadcasts from LANMan servers. |
<domain_name | [1Bh] The name registered by all domain controllers in a domain. |
<domain_name | [1Ch] The name registered by all domain controllers, which can contain up to 25 IP addresses of other domain controllers. This group is used to provide a client with a list of domain controllers that can authenticate its logon request. |
User Names |
|
<user_name | [03h] The name that registers logged-on users for net send operations. |
Machine Names |
|
<computer_name | [00h] The name registered by the Workstation or Redirector service on this computer. |
<computer_name | [03h] The name registered by the Messenger Service on this computer, which is used to issue a net send command to a machine name. |
<computer_name | [+++nnh] The name used by the Network Monitoring agent when it's installed on this computer (nn indicates any hexadecimal value). |
<computer_name | [1Fh] The name registered by the NetDDE service on this computer. |
<computer_name | [20h] The name registered by the Server service on this computer. |
In Windows NT 4.0, you can now specify your own 16th Byte value (in addition to those shown in table 15.1) by using the Internet Group type Static Mapping. Each Internet Group type you define can contain up to 25 IP addresses. For example, you might create an Internet Group of printers called PRINTERS, with a 16th byte of [EFh]. An application could then query WINS for a list of all [EFh] type printers and perform some operation based on the list WINS returns.
When troubleshooting WINS problems, it's important to keep in mind the different 16th Byte values (also called types). If the WINS database becomes corrupted, certain machine functions might be disabled because a 16th Byte entry for the service is missing or has the wrong value. Later, the section "Understanding WINS and DNS Name Resolution" provides assistance in troubleshooting WINS name-resolution problems.
Be sure to back up your WINS.MDB file periodically. Jet database files aren't immune from corruption. WINS attempts to recover a corrupted WINS.MDB file from data stored in .LOG files, but full recovery isn't always successful. Choose
Backup Database from WINS Manager'sMappings menu to create a backup for your WINS.MDB file.You can also enable automatic backup of the WINS database each time the WINS service is stopped. From WINS Manager, choose C
onfiguration from theServer menu, click Advanced, and mark the Backup on Termination check box.
Part of RFCs 1001 and 1002 calls for defining NetBIOS over TCP/IP devices based on the method(s) by which they access NetBIOS services. (The methods are called nodes.) For example, if a Windows NT server doesn't use WINS, all name-resolution requests are via IP broadcasts. In most cases, the broadcast is a name-resolution packet sent to IP broadcast address 255.255.255.255. If the server is configured to use WINS, the server issues a directed request to the WINS database, and then follows with an IP broadcast if the requested name isn't registered with WINS, called h-node resolution. The four sequences of name resolution are as follows:
By default, a Windows host configured with a static IP address and specifying a particular WINS server is an h-node device. If you want to force another node type, you can use DHCP (Dynamic Host Configuration Protocol) to automatically assign client IP addresses and specify the node type.
See "Dynamic Host Configuration Protocol (DHCP)," (Ch 17)
Windows NT provides a variety of built-in tools for troubleshooting network problems. Using Windows NT tools is important because traditional network troubleshooting tools don't always support the NetBIOS implementation of network protocols that NT requires. It's particularly important that the hardware or software protocol analyzers you use recognize the unique nature of Windows NT-specific operations such as DHCP, WINS, and SMBs.
Protocol analyzers are invaluable tools for discovering problems on a network or simply understanding the flow of packets between servers and clients. The advantage of a dedicated (hardware) protocol analyzer is the capability to capture problems from the OSI physical layer up to the application layer. If you have a Token Ring segment that's beaconing-that is, generating an excessive number of beacon tokens because of some problem on the ring-you can see what device is responsible. If your network is experiencing intermittent loss of connectivity, a protocol analyzer might show you that a specific server or client workstation is generating a large number of malformed packets, in which case you can remove and repair the offending device.
The best known hardware-based protocol analyzer is Network General's Expert Sniffer. The Sniffer usually comes bundled with a portable PC and includes a specialized network interface card that's designed to capture and decode packets. Network General also supplies PCMCIA cards and software for use in specific notebook PCs, which let you build your own Sniffer. Network General also produces the Distributed Sniffer, which is a specialized box that lets you plug into and capture packets on multiple segments simultaneously.
Devices such as the Expert Sniffer not only allow you to capture and decode packets, but also provide help with diagnosing problems by analyzing the data and suggesting possible causes for detected problems. Further information on Network General's Sniffer product line is available from http://www.ngc.com/product_info/product_info.html.
Software-only protocol analyzers are less costly than hardware devices but provide less functionality. Examples of popular software-based analyzers are Novell's LANalyzer for Windows (see http://corp.novell.com/market/apr96/mm000109.htm for more information) and Microsoft's Network Monitor, which is part of Microsoft System Management Server (SMS) and now is included with Windows NT Server 4.0. Both applications provide the capability to decode network packets and support various types of pre- and post-packet filtering for several protocols. Network Monitor (NetMon) is ideal for troubleshooting Windows NT networking problems because NetMon is designed to identify the NetBIOS elements specific to Windows NT networks.
All protocol analyzers, regardless of whether they're hardware- or software-based, must be connected to each segment you want to monitor. You need an analyzer agent on each segment of a routed network to capture packets for analysis. The agent is necessary because routers don't forward some packets that may be of interest, such as broadcast packets. Figure 15.3 shows a two-segment network with hardware and software protocol analyzers.
A two-segment network with a hardware and software protocol analyzer.
If you use a software agent such as Novell's LANalyzer, you must make sure that the server is using a network interface card that supports promiscuous mode. In normal operating mode, a NIC captures a packet on the wire only long enough to test whether the destination MAC address is its own. If not, it sends the packet back onto the wire. In promiscuous mode, a NIC captures each packet, regardless of whether the packet is meant for its address, and passes the packet to the analyzer agent. Modern NICs support promiscuous mode, but if you have older NICs, such as the original IBM 4/16 Token Ring card, you must upgrade the NIC.
Promiscuous mode operation, however, can induce at least 30 percent more load on the CPU in the PC where the NIC is installed. As a result, Windows NT 4.0's Network Monitor agent now supports Microsoft's NDIS 4.0 specification. This means that any NIC-even those that don't support promiscuous mode-can capture packets to NetMon, as long as the NIC is using NDIS 4.0 drivers.
The Network Monitor tool that comes with Windows NT Server 4.0 is very handy for troubleshooting Windows NT-related network problems. Because it's a software-based tool, you can use it to view packets going to and from your Windows NT server. The Network Monitor Agent Service comes with Windows NT Server and Workstation 4.0, and must be installed and running before you can use Network Monitor. You need only one device running the Network Monitor agent per routed segment, and that device can be running on either Windows NT Server or Workstation 4.0.
The version of the Network Monitor application that's included with Windows NT Server 4.0 is limited in its capabilities, compared with the version that now ships with Microsoft System Management Server (SMS) 1.2. Specifically, for security reasons, Windows NT Server 4.0 doesn't let you connect to remote NetMon agents running on Windows NT systems across a routed network. You must run the SMS version of NetMon to gain this functionality.
You have two options when installing NetMon components on Windows NT Server 4.0. The first option lets you install just the Network Monitor agent. The second option installs both the NetMon GUI tool and the agent. You choose the agent-only install if you have the SMS version of the NetMon application and want to enable monitoring across a routed network. In this case, the agent-only option is installed on one Windows NT Workstation or Server per segment.
To install the Network Monitor Tools and the Agent Service, follow these steps:
If you're using the NetMon version that comes with Windows NT Server 4.0, you can trace packets flowing to and from the server on which you've installed the NetMon application. To use the NetMon application, follow these steps:
Network Monitor's window on initial startup.
Connecting to the Network Monitor Agent.
Capturing packets with Network Monitor.
Viewing all packets captured in the Capture Summary window.
NetMon keeps a database of names that correspond to a given MAC address. You can view the entries in this database by choosing
Addresses from the Capture window'sCapture menu or from the Capture Summary window'sDisplay menu.In figure 15.9, the address database contains some default entries for broadcasts on Ethernet, Token Ring, and FDDI, as well as a special MAC address used by NetBIOS. Also, a Windows NT Server called NTSERVER1 and a Windows 3.1 client called QIEBJMYC have been discovered and automatically added to the database. In the Address Database dialog, you can click Save to save these entries to the default address database, DEFAULT.ADR.
Viewing NetMon's address database.
The default filter expressions of the Display Filter dialog.
Setting up a display filter to show only IP packets.
Viewing detailed information on a captured packet.
A plus symbol to the left of each layer item indicates that there is more detail available under that heading. Double-click one such entry to display detailed information for that portion of the frame. Figure 15.13 shows an IP layer entry that has been expanded to show all the fields of the IP header, including source and destination IP addresses for that packet, and the packet's size. The content of the selected location of the packet appears in hexadecimal format in the pane at the bottom of the Capture Detail window.
Viewing packet detail within the IP header.
Network Monitor is useful in troubleshooting various network-related problems, such as name resolution by a WINS server. You can use NetMon to capture packets coming into the WINServer to determine whether a client's name-resolution requests reach the server, and whether the server responds to the requests. By using NetMon's trace capability, you can verify that the client makes a WINS request to the server and the server returns a packet indicating that the name wasn't found. You then use WINS Manager to determine whether the name exists in the server's WINS database or whether the WINS database is corrupted.
Windows NT's Performance Monitor (PerfMon) is a valuable tool for monitoring the operation of Windows NT servers and workstations, including their network components. PerfMon lets you monitor network interfaces on a server to determine bandwidth usage, rates of errors and broadcasts, and protocol-specific counters. Figure 15.14 shows PerfMon's Add to Chart dialog listing some of the counters available for the Network Segment object.
See "Using Performance Monitor," (Ch 14)
Performance Monitor's Network Segment counters.
The % Network Utilization counter gives a running snapshot of current network usage on the segment you're monitoring. This counter is a valuable tool for performing a quick analysis of a segment where you suspect problems are occurring. You can launch PerfMon and do a quick check without having to install a protocol analyzer or software agent.
You need to install the SNMP (Simple Network Management Protocol) service to obtain all the available TCP/IP and/or IPX statistics for your network interface. Without SNMP, PerfMon can't see some of the network objects. To install SNMP service, follow these steps:
Installing the SNMP service from the Select Network Service dialog.
Setting SNMP Agent configuration options.
Setting Security properties for the SNMP service.
As part of the installation of the SNMP service, Microsoft provides a number of Management Information Bases (MIBs). These MIBs contain the various attributes that the SNMP service tracks. SNMP managers monitor the MIB attributes of servers, routers, and other SNMP-compliant network devices.
If you're familiar with the SNMP MIB specification, Windows NT provides two trees in the private MIB section. 1.3.6.1.4.1.77 is the LANManager tree that provides information on shares, sessions, and users. 1.3.6.1.4.1.311 is the Microsoft tree that provides statistics on WINS and DHCP. A tree in the MIB, as represented by the Object ID (OID) 1.3.6.1.4.1.77, is really an "address" where an SNMP manager that's connected to Windows NT Server's SNMP service can attach to find SNMP information about a specific attribute on the server. Under each tree are leaf objects corresponding to individual characteristics for that device. For example, under the 1.3.6.1.4.1.311 tree for WINS and DHCP, an SNMP Manager can find what entries are contained in the WINS database on that server, or who the WINS service's replication partners are.
After installing the SNMP service on your Windows NT server, you can use PerfMon to gather IP- and/or IPX-related statistics about your system. The most useful application of PerfMon as a network troubleshooting tool is tracking protocol-related information over time with PerfMon's logging function. By using the IP object in PerfMon, for example, you can track datagram errors, packets received, or packets discarded over time (see fig. 15.19).
PerfMon's Add to Chart dialog showing IP-related counters.
PerfMon also includes counters for ICMP, TCP, and UDP objects that provide similar information to IP. Tracking these counters over time provides valuable information about network performance on your Windows NT server, especially when you must troubleshoot TCP/IP or IPX network problems.
Windows NT 4.0 and the Windows NT Resource Kit provide many useful command-line tools for troubleshooting network problems. The following sections describe the most important tools included with Windows NT Server 4.0 and the additional command-line utilities provided in the Windows NT Resource Kit.
You can use the command-line tools that come with Windows NT 4.0 to help solve many basic networking problems. Most of the tools are for TCP/IP and are likely to be familiar to UNIX users.
The arp (address resolution protocol) command lets you view the current contents of the arp cache on a server or workstation. arp -a displays the contents of the arp cache; arp -d and arp -s let you manually remove and add entries to the arp cache. With ping, you can use arp to determine whether a device is communicating on the network. If you ping a device in question, you should see a corresponding entry in the arp cache of either the device you're pinging, or that of the default gateway if the device isn't on your local subnet (see fig. 15.20).
Using the arp command to view the arp cache.
The hostname command returns the name of the system on which the command is executed. The name returned is the name specified in the DNS setup section of the TCP/IP configuration of the system, rather than the NetBIOS name.
The ipconfig command returns all the current TCP/IP, DNS, and WINS information for the system. Use ipconfig /all to display all the information, and just ipconfig to obtain abbreviated information (see fig. 15.21).
Viewing the output of the ipconfig command.
The ipconfig command is also a quick way to get the MAC address of the NIC installed in the device. If you're using DHCP, you can also use the ipconfig command to renew a DHCP address reservation, or release an address on a DHCP client using the /renew and /release parameters.
The nbtstat command is one of the most useful Windows NT networking tools, because it provides various information about NetBIOS names and their addresses. For example, if you know the NetBIOS name of a workstation and want to know its IP address, follow these steps:
You also can use the nbtstat -A ip_address command to determine what machine is registered to a given IP address (see fig. 15.22). Note that this command requires an uppercase A parameter. When you issue this command, the server or workstation sends a name request to the IP address of the primary WINS server specified in the issuing device's TCP/IP WINS configuration page. The returned information is the contents of the WINS database for ip_address. This command is useful if you're trying to troubleshoot WINS problems.
Viewing the output of the nbtstat -A command.
The netstat command performs many of the same functions as its UNIX counterpart. The netstat -a command displays all current TCP and UDP connections from the issuing device to other devices on the network, as well as the source and destination service ports, and-in the case of TCP-the current state of the connection (for instance, Established or Time-Wait). You also can use netstat -r to post a listing of the routing table on a given machine (see fig. 15.23).
Using netstat -r to view the routing table of a Windows NT server.
The netstat -e command gives you statistics on your network interface. When combined with an interval parameter-for example, netstat -e 10-the following information is updated every 10 seconds:
Interface Statistics Received Sent Bytes 184763 125248 Unicast Packets 304 437 Non-unicast Packets 1419 1419 Discards 0 0 Errors 0 0 Unknown Protocols 313
The preceding information is useful if you're troubleshooting suspected network problems and want to determine whether the network interface is generating errors.
The ping command is widely used for testing connectivity. This command sends an ICMP echo packet to the host or IP address you specify on the command line. For example, ping 200.200.1.1 sends an echo packet to IP address 200.200.1.1. If ping is successful, you see a series of replies similar to the following:
Reply from 200.200.1.1: bytes=32 time=10ms TTL=32 Reply from 200.200.1.1: bytes=32 time=10ms TTL=32 Reply from 200.200.1.1: bytes=32 time=10ms TTL=32 Reply from 200.200.1.1: bytes=32 time=10ms TTL=32
If ping fails, you receive the following message:
Request timed out.
You can test whether a server's TCP/IP subsystem is working correctly with the ping 127.0.0.1 command, also known as the loopback address. This is the localhost address, or the default hostname for that machine; this address and hostname is installed by default when you install Windows NT's TCP/IP stack. A successful ping indicates that this virtual address is alive, and that your TCP/IP stack is functioning correctly.
The route utility lists a server's TCP/IP routing table and can add or delete static routes. You also can add persistent (static) routes that are maintained through shutdown and startup. To view the current routing table, type route print.
You can use the -p parameter to create a persistent route. The following example shows how to add a static route to a remote network and make it persistent:
route -p add 192.168.20.0 MASK 255.255.255.0 200.200.1.255 1
The trailing 1 indicates that the remote network is one router hop (one pass through a router) away. The example routes to the remote network 192.168.20.0 using the gateway/router address of 200.200.1.255. The remote network has a subnet mask of 255.255.255.0.
Persistent routes are stored in the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\PersistentRoutes of the Registry.
The tracert command lets you trace the path to a destination IP address, identifying all the intermediate hops between the source and destination. You can use the tracert command-a very powerful tool for determining how packets are traversing your network-to troubleshoot routing loops or down routers, as well as discover timeout problems across the network.
The tracert command uses ICMP to find the path to the end station you specify. The following example, tracert EndStation, traces a path to a client called EndStation at IP address 172.16.12.1:
Tracing route to EndStation [172.16.12.1] over a maximum of 30 hops: 1 181 ms 130 ms 130 ms 172.16.4.254 2 160 ms 131 ms 120 ms routera.mycompany.com [172.16.5.254] 3 151 ms 120 ms 120 ms routerb.mycompany.com [172.16.6.254] 4 160 ms 140 ms 140 ms 172.16.8.254 5 161 ms 140 ms 140 ms routerc.mycompany.com [172.16.9.254] 6 170 ms 141 ms 130 ms EndStation [172.16.12.1] Trace complete.
In this example, the path to the EndStation client is six hops from the device where the command is initiated. The first hop is to the default gateway (172.16.4.254) on the subnet where the command was issued. Along the way, tracert does a DNS reverse-address lookup of each hop (unless you specify the -d parameter) to resolve IP addresses of intermediate hops to host names. If your router interfaces aren't configured in DNS, all you see are the addresses in lines 1 and 4 of the example.
A DNS reverse lookup is exactly as it sounds. Rather than resolve a known host name to an IP address, a reverse lookup resolves a known IP address to a host name.
In the following example, the EndStation client is unreachable from the source. The trace stops after the third hop. This could mean that either the end station is down, or the destination subnet isn't accessible from the source. The latter problem can be due to router problems or to an intermediate route filter of some type. The best way to approach solving these kinds of problems is to determine what part of the path does work, and narrow the possible suspects as you go. A good troubleshooting technique with tracert is to connect to the last successful step along the path-in this case, routerb.mycompany.com-and determine whether you can get to the destination from there. If not, your problem is likely to be in a destination device or an intermediate router. You might need to run another trace from routerb to determine whether intermediate hops to the destination are failing.
Tracing route to 172.16.80.1 over a maximum of 30 hops 1 140 ms 120 ms 130 ms 172.16.4.254 2 131 ms 120 ms 120 ms routera.mycompany.com [172.16.5.254] 3 routerb.mycompany.com [172.16.6.254] reports: [ccc]Destination host unreachable. Trace complete.
The Windows NT Resource Kit includes several useful tools for troubleshooting network problems. Many are easy-to-use GUI-based tools, but others are cryptic, not very well documented, and require trial-and-error experimentation to use correctly. Microsoft provides product support (including bug fixes) for the Resource Kit utilities, but doesn't respond to requests for enhancements. You also can send questions by e-mail to rkinput@microsoft.com; someone at Microsoft might get back to you, if only to tell you that the information you need is available somewhere else.
The Windows NT 4.0 Resource Kit wasn't available when this book was written. The descriptions that follow apply to the tools included in the Windows NT 3.51 Resource Kit, few of which worked with Windows NT Server 4.0.
Following is a description of the utilities in the Resource Kit that are most useful for network troubleshooting.
Browmon.exe is the Browser Monitor, the 3.51 version of which is incompatible with Windows NT 4.0. BrowMon is very useful when you need to troubleshoot browser problems. You can check the contents of the Master Browser on each subnet in question to determine whether your missing device is actually contained in the browser list for that segment.
When you run BrowMon, you see only browser information for the local network segment. If you want to know what's contained on the Domain Master Browser (almost always the Primary Domain Controller, or PDC), you must run BrowMon from a machine on the same segment as the PDC.
Browstat.exe is the Browser Status command-line utility that's useful for retrieving general browser information, such as statistics and domain information. The 3.51 version of BrowStat is incompatible with Windows NT 4.0. You can also use BrowStat to force a browser election, force a master browser to stop, and find out if you have any Windows for Workgroups workstations acting as Master Browsers on a network segment.
Dommon.exe, the Domain Monitor utility, displays the current status of a given domain's Domain Controller, including whether its Security Accounts Manager (SAM) databases are synchronized. DomMon also shows the current trust status of any domains with which the current domain has a trust relationship. The 3.51 version of DomMon is incompatible with Windows NT 4.0.
DomMon is useful for troubleshooting problems with trust relationships, because it allows you to see which remote domain controller the current domain uses for the trust. From this point, you can verify connectivity between the two servers as part of your troubleshooting process.
Getmac.exe is a simple utility that returns the MAC address of the machine on which it's run. The 3.51 version of GetMac works with Windows NT 4.0. GetMac is useful if you're having problems with ARP and need to know MAC addresses. GetMac returns the MAC address of each network protocol loaded on your machine, similar to the following:
E:\ getmac Transport Address Transport Name ----------------- -------------- 20-4C-4F-4F-50-20 \Device\NetBT_NDISLoop1 00-00-00-00-00-00 \Device\NetBT_NdisWan5 20-4C-4F-4F-50-20 \Device\Nbf_NDISLoop1 20-4C-4F-4F-50-20 \Device\NwlnkNb
Netwatch.exe is a GUI-based utility, similar to Windows 95's Net Watcher, that lets you monitor user connections to shares on servers and clients. The 3.51 version of NetWatch works with Windows NT 4.0. As figure 15.24 shows, you also can see what files a user has open on a given share.
Using the NetWatch utility to view users logged on to a Windows NT Server 4.0 share.
Windows NT doesn't have the NetWare 3.x concept of user logon connections into each server; Windows NT attaches to domain resources as needed. This utility, however, tells you who's connected to which share on a given server or client, and which files they have open.
See "Using Net Watcher Remotely," (Ch 10)
Nslookup.exe serves the same function as its UNIX equivalent. The 3.51 version of Nslookup.exe works with Windows NT 4.0. You can use NSLookup to query DNS servers to determine the IP address of a specified host name, or the host name of a specified IP address. The nslookup command sends the request to the address and subdomains defined in the DNS configuration portion of Windows NT's TCP/IP configuration screen (see fig. 15.25). nslookup takes no parameters. When you enter the command, you're placed at the > prompt, where you enter the name or address you want to resolve (see fig. 15.26).
Setting DNS Configuration options in the Microsoft TCP/IP Properties sheet.
Using nslookup to resolve DNS names and addresses.
The Smbtrace.exe command-line utility is similar to the UNIX etherfind command, except that smbtrace tracks Server Message Block packets only. The 3.51 version of Smbtrace.exe works with Windows NT 4.0. SMBTrace is a real-time packet tracer that shows the SMB packets that are flowing to and from your system, which is useful for understanding the flow of SMB packets during file transfers and print jobs. By default, smbtrace captures incoming packets only. The /slow parameter captures both incoming and outgoing, and the /rdr parameter captures only outgoing packets. You can change the amount of information smbtrace displays with each packet by using the /verbosity:n parameter, where n is a number between 1 and 5, 5 being most verbose.
The Windows NT IP configuration tool, Wntipcfg.exe, lists all the TCP/IP configuration information for the system on which WNTIPcfg runs. WNTIPcfg provides the same information as ipconfig -a in a graphical format. The 3.51 version of DomMon is incompatible with Windows NT 4.0.
In addition to the tools described in the preceding sections, the following resources also include network troubleshooting advice and tools:
You can find useful NT-related information in many other places on the Internet. Just choose your favorite search engine and search on Windows NT.
Windows NT Server is similar to other operating systems when troubleshooting basic network problems. If there are connectivity problems, you check the cabling, the network interface card, and the network protocol configuration. Windows NT, however, includes a number of additional network-related components (such as browsing, trusts, and WINS) that can compound basic network problems. An understanding of some of the common problems related to Windows NT's special networking features helps you become a more accomplished network administrator. The following sections explore some of the more common problems and suggest possible solutions.
Connectivity problems manifest themselves in different ways. In Windows NT, a connectivity problem can cause trust relationships to fail or cause drive mappings to time out.
You were introduced to a number of tools in the preceding sections for testing connectivity. In the following sections, the utilities are applied to help solve connectivity problems that arise in Windows NT environments.
Utilities such as ping allow you to test connectivity from point A to point B. The ping utility may succeed, but success verifies only that TCP/IP is working. If you must verify Windows NT Server's file, print, and application service availability, you must be able to verify connectivity at the NetBIOS level.
The net view command is the NetBIOS equivalent of ping. A net view \\servername command might result in the following message:
System error 53 has occurred. The network path was not found.
This message usually means that either the servername system is down, or the device's name can't be resolved. You may find the case where ping succeeds but net view fails. An example of such a situation is when a NetBIOS device, such as a server, has found a name conflict on startup. While the server was shut down, WINS may have registered another name at the server's IP address, or the same server name may have been registered at a different network address. You can use the nbtstat -n command to determine whether there's a conflict. In either case, the network subsystem might start, but the NetBIOS services don't. You can check for this situation by searching the system log with the Event Viewer application on the device in question.
If you receive system error 53 and have verified that the system's NetBIOS subsystem isn't in conflict, you likely have a name-resolution problem. In this case, verify on a TCP/IP network that you have an entry in WINS or an LMHosts file to resolve the name in question.
If you suspect your connectivity problems are related to routing-that is, you've verified that both source and destination are up and can access devices on their local segment-use tracert to determine where in the path you have a problem.
If you're in a networked environment where each segment has multiple, redundant routes to reach a specified destination, you should be aware of an idiosyncrasy in Windows NT's TCP/IP stack. Windows NT Server 4.0's Advanced IP Addressing dialog provides the capability to specify multiple default gateways to reach remote segments (see fig. 15.27). If the first gateway in the list fails, however, any subsequent User Datagram Protocol (UDP) packets fail to search the list for the next gateway. When the first TCP packet finds the primary gateway down, the packet goes down the list to the next gateway and makes it the active gateway. Thus, subsequent UDP packets use the new gateway to reach their destination. You may never notice use of an alternative gateway unless the primary gateway is down when you're first powering up a system.
Setting multiple default gateways with the Advanced IP Addressing dialog.
The Netlogon Service, which is responsible for domain authentication, uses UDP to authenticate a client and user to a domain. As a result, when a server that is a backup domain controller (BDC) starts up with its primary default gateway down, the BDC can't contact other servers that aren't on the same physical network segment-that is, across a router. The result is the following message:
A Domain Controller could not be contacted
In this case, until the first TCP packet goes out, the BDC can't authenticate-and therefore synchronize-to the domain. Although this is a specialized problem of redundant routing, it's significant, especially on the workstation side.
Browsing is a feature unique to Windows NT and its predecessor, LAN Manager. Browsing is solely a function of NetBIOS and slightly parallels Novell's SAP service announcements. Troubleshooting browsing problems, especially in a NetBEUI or IPX-NetBIOS environment, is downright difficult. You're limited to the BrowMon and BrowStat tools (discussed earlier in this chapter). Troubleshooting browsing problems is a bit better in the TCP/IP environment, where browsing is broken up by logical subnets and browser advertisements are broadcast-based, thus not forwarded by the routers.
The best way to troubleshoot browsing is first to understand the browsing process. A domain environment has three types of browsers, as follows:
Following is a step-by-step description of the election process to determine a new Master Browser:
Browser operations use NetBIOS mailslots to perform tasks such as announcements, elections, and updates. Windows NT Server's mailslots are second class. First-class mailslots provide guaranteed delivery; second-class mailslots are connectionless, so delivery of messages isn't guaranteed.
If you're having browser problems, the first thing to check with BrowMon is the contents of the Master Browser on your network segment. If the computer in question isn't in the list, a possible cause is a Windows for Workgroups client on the segment that has the MaintainServerList parameter of its SYSTEM.INI [386Enh] section set to something other than No.
Another possible problem is that the browser service on a Windows NT server or workstation isn't running. You can check this with Windows NT's Control Panel's Services tool, or issue the net start command to see which services are running.
Finally, you can prevent a Windows NT workstation or server from announcing itself to the Master Browser by issuing the following command:
net config server /hidden:yes
If someone has set the /hidden parameter, you don't see the server on the browser, but you can connect to the server by *using its UNC name.
By default, Windows NT Server doesn't support dynamic routing. That is, if you have two NICs in a server, you can't send routing updates between the two NICs. You can set up routing between the two cards by selecting the Enable IP Forwarding check box in the Routing page of the Microsoft TCP/IP Properties sheet (see fig. 15.28), accessed from Control Panel's Network tool.
Enabling IP routing in the Routing page.
Microsoft includes with Windows NT Server 4.0 support for dynamic routing with the RIP for NWLink IPX/SPX service and RIP for Internet Protocol Service. Unless you run these services, you must configure routing with the static route command described earlier in the chapter. Most problems with routing in Windows NT relate to this static routing concept. Specifically, if you're trying to communicate from a segment connected to a NIC in a Windows NT server to another segment connected to a second NIC, also installed in that server, you need a static route.
In the example shown in figure 15.29, if you're trying to communicate from Client A on Network B to Server B on Network C, Windows NT Server C needs a static route to tell traffic how to get to Network C. The persistent route statement is
route -p add 172.16.42.0 MASK 255.255.255.0 172.16.40.2 1
where 172.16.42.0 is the destination network Client A is trying to get to, and 172.16.40.2 is the router interface on the far end of Network A that forwards the packet to the correct network.
An example of a configuration that requires a static route.
Trust relationships allow resources and user accounts to be shared between domains. The major network problems that arise as a result of trusts are in a WAN using TCP/IP. Domain controllers use WINS to determine names and IP addresses of trusted domain controllers in remote domains. As a result, if there are problems with WINS, trust relationships break down.
If you're replicating your WINS databases so that all domain controllers in each domain can resolve each other, establishing and maintaining a trust relationship should be easy. Domain controllers, by default, register their domain names in WINS using the [1Ch] 16th Byte type. However, if this replication has failed or has become corrupted, you can statically map domains in WINS so that each side of the trust can resolve the other-in effect, statically adding the [1Ch] entries. Figure 15.30 shows an example of static mapping of trust domains in the Static Mappings dialog of WINS Manager. To access the WINS Manager, use the Start menu and choose Programs, Administrative Tools, and then WINS Manager. When you click Add Mappings in the Static Mappings dialog, you see a place to type your static entries, and a number of choices for entry type (see fig. 15.31).
Displaying WINS static mappings with WINS Manager.
Setting static mapping options in the Add Static Mappings dialog.
If you're establishing a trust relationship and dynamic registrations in WINS don't seem to be functioning correctly, you can statically map both the remote domain and its domain controllers in your WINS database. Doing so guarantees that your domain controller always has the correct information on how to reach the remote domain, even if the remote domain's WINS database is corrupted and doesn't provide the proper dynamic registrations to other domain controllers.
To statically map a remote domain in your WINS database, follow these steps:
Adding a Domain Name type static mapping from the Add Static Mappings dialog.
You can verify that your static entries have been entered into the WINS
database by choosing Show Database from the Mappings
menu in WINS Manager. Figure 15.33 shows the contents of a WINS database
after adding the static domain mappings. This example shows a remote domain
called TrustDomain, which contains a server called TrustDomain_PDC. The
[1Eh] value indicates the Group entry for the remote domain. The [1Ch]
value indicates the Internet Group entry, and the three Unique type values
([00h], [03h], and [20h]) are for the PDC itself. If you have a two-way
trust between domains, create these static WINS mappings so that they point
to each domain on either side of the trust.
Creating a static mapping set for a remote domain.
As noted at the beginning of this chapter, name resolution is the source of many networking problems with Windows NT Server. This is especially true if you're using the TCP/IP stack, because you may also have to contend with DNS as a potential name service for NetBIOS name lookups. Windows NT Server offers the following places where you can store names for resolution in a TCP/IP environment:
Depending on the network function being performed, the path that name resolution follows varies. When you issue a net view command to a servername, for example, Windows NT first determines the node type of the device (h-node, p-node, m-node, or b-node). Based on the determination, NetBIOS name resolution follows these steps:
Setting advanced configuration options for name resolution.
The process Windows NT uses for name resolution when performing non-NetBIOS operations varies. For Winsock-based (Winsock is the Windows Sockets standard) operations such as ping, Telnet, FTP, or any other function or service that doesn't require name resolution of NetBIOS resources, Windows NT uses the following steps:
Solving name-resolution problems requires an understanding of the process, and then troubleshooting each step of the process. A good tool for determining where the flow stops is a protocol analyzer, which also tells you whether significant delays in name resolution are occurring. Such delays can have the effect of breaking an application that depends on the timeliness of name resolution. If you can detect delays with a protocol analyzer, you can move name entries to guarantee that they occur more quickly.
For example, a DNS lookup for a given host may take more time than the calling application allows. In that case, you might move that hostname address mapping into the local %systemroot%\system32\drivers\etc\hosts file for faster resolution. Alternatively, if WINS is taking too long to resolve a name, or you experience delays in reaching a WINS server, you can increase the timeout of the NetBIOS name cache on your server. By default, name-resolution responses from WINS are cached in the name cache for 10 minutes. If your environment is fairly stable-that is, Windows NT devices don't change addresses often-you can increase the value of the name cache timeout. NetBIOS name resolution looks to the name cache first, so you go to WINS less often for name resolution.
You can increase the timeout value for the NetBIOS name cache by editing the Registry on your Windows NT Server. The Cache Timeout value is stored in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NetBT\Parameters\CacheTimeout. The value in this key is in milliseconds, and defaults to 600000 (600 seconds). You can change the value from 60000 to about 4 million milliseconds. Change the value in small increments either direction, depending on your network environment. Too small a value results in excessive requests to WINS. If you set the value too large, devices whose addresses have changed might never be found.
This chapter discussed how the different network protocols that Windows NT supports for file, print, and application services affect network troubleshooting. NetBEUI and IPX are broadcast-based protocols that require a bridged network or special router forwarding and aren't very scalable. Windows NT's NetBIOS-over-TCP/IP support, also called NetBT, uses a centralized name service (WINS) to avoid the need to broadcast between subnets. Using TCP/IP and NetBT simplifies network troubleshooting.
The chapter also described the utilities that come with Windows NT 4.0 for interrogating and checking network-related services, such as ping, nbtstat, and tracert, and programs included in the Windows NT Resource Kit.
The chapter concluded with a discussion of common Windows NT-specific network problems, such as browsing and name resolution, and how you can use the tools and utilities to aid troubleshooting network problems.
The following chapters contain information related to network troubleshooting:
© 1996, QUE Corporation, an imprint of Macmillan Publishing USA, a Simon and Schuster Company.