DeviceMaster Latency Benchmark

Thank you for your interest in learning more about state-of-the-art serial device connectivity.

This White Paper provides a snapshot of network serial device servers (a.k.a. - serial port servers, edge computers, network appliances, serial hubs) and how the leading products in this segment perform when connecting serial devices to application software using the Ethernet as the communication medium. This report focuses on LATENCY, the cause for it and the impact that it has on the speed of processing data across a local-area network. We believe the test is fair and equitable. And we believe that the results accurately portray the performance of the various products that are currently the market leaders in this technology segment. We want you to be totally convinced that the products perform as described in this report - so we have provided you with everything you need to recreate the benchmark yourself (except of course the hardware). If you run your own benchmark and have any questions, we would like to hear from you. Please contact David Boldt (benchmark co-author) at david.boldt@comtrol.com or 763-494-4100.

JUST FOR FUN … Once you have seen the benchmark results in tabular form, you might want to have some fun by watching the competitors fight it out on the race-track. Since the benchmark measures elapsed time, we thought a drag race would be in order - where the first to the finish line is the winner.

EXECUTIVE SUMMARY

More than 20 years ago, engineers at Comtrol responded to a request for an expansion card to be used with the "new" personal computer so that it could connect multiple peripherals using serial communications. The result was the first "multiport serial concentrator card" for personal computers, which has since been widely adopted as the standard for connecting electro-mechanical devices using serial communications ("serial devices"). This traditional approach of connecting serial devices to a PC using a multiport card has a well-known set of limitations:

• Need for specialized cabling (including cost and time required to accomplish the installation)
• Physical distance limitation of 50 feet between a PC and an RS-232 serial device
• Serial device operational dependency on continued availability of the host PC (application server)
• Personal computer cost of ownership (e.g. acquisition, maintenance, support)

In the mid-1990s, Comtrol learned of the pending obsolescence of the PC ISA expansion bus that would force user migration to the (then) new PCI expansion bus. Anticipating more forced migration in the future (which has happened with the Universal Serial Bus and then "universal" PCI expansion bus), Comtrol searched for an alternative high-speed communication bus that might be more stable over the long term.

At the time, Ethernet was already installed within most major organizations across virtually all market sectors. Since Ethernet was already in place, using it as a data communications medium would eliminate the need for costly serial cabling. Plus, Ethernet also solved the 50-foot distance limitation of RS-232 serial communications, enabling deployment of a serial device anywhere an Ethernet connection was available. Finally, host-PC dependency could be more effectively managed by installing a backup version of the application software on a different networked PC. You could then switch from one to the other if the primary PC failed - keeping the serial device running!

Having identified Ethernet as the optimum communications bus, all that was missing was a cost-effective, reliable platform to connect the serial devices and handle network communication. In 1997 Comtrol developed the industry's first "serial device server" (InterChangeVS™). This groundbreaking innovation led the way for more powerful successor device servers, culminating with today's U.S. market leader* - DeviceMaster!

Comtrol device servers have now been available for seven years. While the adoption rate is growing, the traditional approach of using PCs with multiport serial expansion cards remains the predominant device connectivity technology.

Comtrol believes that a major barrier of device server adoption is the presumed latency associated with network communications. Many design engineers assume that the overhead introduced by the Ethernet and its network communications protocol processing would decrease communication speed across the network, making it an unrealistic substitute technology for PC/Device direct connections.

In an effort to provide quantifiable evidence of network-induced latency, Comtrol established a benchmark study called the Serial Shoot-Out. The test sent data from a PC, over an Ethernet network to a serial device server, and back again. The elapsed time of each test was recorded. The test was repeated twice, running 10,000 iterations each, and then the average was reported as the product's observed latency. To provide comparison to traditional device connectivity methods, the test was also run on a native serial port (located on a PC motherboard). Finally, to remove any doubt about the validity of the test, Comtrol engaged a third-party testing service to run the benchmark, which independently verified the results. As shown below, the Comtrol and test service results were very similar, indicating that the benchmark is valid and repeatable.

All products were tested using industry-standard TCP/IP network communication protocol so the results are directly comparable. DeviceMaster products, however, were also tested using Rapid Transport Service™ (RTS), Comtrol's proprietary, patent-pending protocol that is designed to maximize serial data communications over Ethernet. Interestingly, when the DeviceMaster was run with RTS, it produced nearly the same latency as the directly connected serial port - and in the independent test lab report, actually BEAT the native serial port latency!

The results speak for themselves … all of the Comtrol products tested significantly out-performed the competition in both TCP/IP and RTS communication modes!

Using the latency and computed throughput generated by this test, design engineers now have a way to determine if deploying a serial device server as a replacement for a PC/Serial Card will meet performance requirements for a given application. It must be emphasized, however, that actual results may deviate from the test results primarily due to factors beyond the scope of this study (such as network traffic). The only way to know for certain if this alternate technology will work is to try it - and Comtrol has no-risk evaluation units for qualified customers!

There is one final observation about the test results.

On October 16, 2002, Digi International issued a news release declaring its Digi One IA as the fastest performer in a similar network-latency benchmark test. In comparing the results of the two tests, we find that the tested products produced similar performance with the exception of the Digi One IA. To date, Digi has not shared its test programs with any third parties, so Comtrol is unable to precisely determine why the results are so significantly different. Note that both the Comtrol and Digi test were performed by the same independent testing service, and they too are at a loss to explain the performance differential. Unlike Digi, Comtrol has provided all of the test parameters and the actual test software so that anyone (including Digi) can replicate this benchmark. We are confident with the results and confident that DeviceMaster is the unequalled serial device server performance leader!

Thank you for your interest in Comtrol and DeviceMaster.

SERIAL DEVICE CONNECTIVITY OVERVIEW

Personal computers equipped with serial multiport expansion cards remain the most prevalent method of connecting and communicating with electro-mechanical devices. Figure 1 depicts this configuration with a multiport card (RocketPort^®) installed in the PC, communicating with an application through the multiport driver software and operating system.

The "quality of service" provided by a serial multiport expansion card is directly related to the completeness of its driver software. The user application software is written to use specific serial communications features and functions supported by the operating system running on the PC. The mutliport card device driver implements the features and functions provided by the operating system, making them available on the additional serial ports added by the multiport expansion card. Accordingly, each driver must be customized to support the functions provided by each unique operating system, resulting in the quality of service relationship between software driver and card.

Figure 2 shows the alternate configuration using the Ethernet as the data communications bus. In this mode, the driver has a dual role. Not only does it have to interface with the operating system to provide the serial port features and functions, but it also must communicate over the network to the physical serial ports. For this reason, the driver for a serial device server is divided into two modules - one module runs on the host PC and another runs on the serial device server. The modules then communicate with each other using a network data transfer protocol. Therefore, in addition to supporting the features/functions of the operating system, these drivers must also make optimum use of network communications - which adds yet another element into the overall quality of service equation.

NETWORK COMMUNICATION PROTOCOL

Three characteristics define a network: topology (physical layout of connected computers), architecture (peer-to-peer or client/server), and protocol. Protocol defines a common set of rules and signals that are used by attached computers for communication. TCP/IP is undoubtedly the most popular protocol and the one that is prevalent in this market.

To promote compliance with the "rules", a model, known as the OSI (Open Systems Interconnection), was created to define a framework of network communication software. This OSI framework consists of seven "layers". Communication is passed from one layer to the next, starting at the application layer and proceeding down the "stack" to the bottom (physical) layer, over the channel and then back up the stack again to the receiving application.

TCP/IP protocol operates slightly differently. TCP/IP replaces four OSI layers with two new layers (TCP and IP) that generally follow the OSI processing schema. TCP/IP also adds routing and error checking capabilities that can increase processing time (and latency) on noisy or heavily loaded networks. Using TCP/IP, the PC-based driver module and the serial device server driver module communicate to transfer data across the physical layer. Using TCP/IP requires that each network device receive an IP (internet protocol) address so that it can be identified on the network. The complexities of the protocol and increased processing due to error checking can increase latency. As a result, network design engineers must consider the minimum tolerable latency of data communications and offset that against the overhead associated with the use of standard network protocols.

NETWORK COMMUNICATIONS LATENCY

In networking and serial communications, latency is defined as the DELAY (measured in time) associated with transferring a packet of data from a source to a destination. In applications using multiport serial cards, latency was rarely a major consideration because the driver and internal high-speed communication bus within the PC imposed only a few milliseconds of latency (delay). However, as we have seen, when a shared resource such as the Ethernet is used as the communication bus, external factors can impact performance and introduce latency. Factors such as protocol handling, network traffic, poorly written "COM port redirector" driver software, and network hubs/routers can introduce additional processing overhead that could potentially add hundreds of milliseconds to the time required for the data to reach its destination.

While a delay of fractional seconds seems irrelevant, in some applications, such as material handling, satellite communications and real-time device monitoring, this small delay can have a significant negative impact on the automation process. Here is a brief example:

Consider a package handling system. The package sits on a moving conveyor. It passes a scanner that reads a bar code on the package. Captured bar code data is transmitted from the scanner's serial port to a PC running a decoder and package routing application program. The system has only 30 milliseconds to analyze the bar code, determine the destination and then transmit a routing instruction back to the conveyor. If a serial device server and Ethernet connection is used instead of a direct PC connection, and this new approach introduces 50 ms of latency to the overall throughput, the package could pass the conveyor routing point before the routing instructions are received. To fix this problem, the conveyor speed must be reduced. But running the conveyor slower reduces its output and efficiency. Clearly this is not a desirable solution!

When the benefits of serial device servers are desirable, but the latency introduced by the TCP/IP network protocol is a problem, a different approach is needed. Comtrol has developed that alternative. Rapid Transport Service™ (RTS) is a proprietary, patent-pending network protocol developed by Comtrol that uses a modified protocol technique, augmented by special software. The result is a "low overhead" protocol that is simple to configure and that produces outstanding low-latency results. But due to modifications made to the protocol, RTS cannot support wide-area networking or routing functions. Therefore, RTS requires that the host PC server and the serial device server reside on the same Ethernet segment. While somewhat limited for WAN use, RTS can deliver the benefits of remote serial ports in the right situations.

SERIAL SHOOTOUT

Comtrol is committed to driver quality of service. Known for driver excellence with its Hostess and RocketPort (multiport) cards, Comtrol engineers accepted the challenge to develop the most sophisticated, high-performance, serial device server driver in the industry. Part of the development process was the creation of a performance benchmark, which would measure the latency of Comtrol products as compared to native PC serial ports and competitive serial device server products.

The benchmark test sends a packet, consisting of one byte of data, over the Ethernet using either TCP/IP or the Comtrol RTS protocol, to a remote serial device server where the data is echoed back to the originating PC. The echoing of data in the tests is accomplished using a loop-back plug that connects the RS-232 Tx data pin to the Rx data pin. Use of a loop-back plug instead of a second computer to echo the data eliminates questionable results that might be artificially introduced by a second computer (since routing data through an intermediate computer adds an unwarranted level of complexity to the process). Since many manufacturers supply a loop-back plug, it is far simpler to set up such a test.

This test provides a realistic approximation of performance under simulated real-world conditions and allows for easy replication with commonly available PC equipment. Plus, the configurable parameters and simplified setup of this test make it an invaluable tool for users and integrators looking to match the right device server product to timing-critical applications. The following hardware was used for the test.

After defining the path that the data will take and the equipment specifications for the "test track", a specialized computer program is required to execute the benchmark. The program runs on the host PC, generates the test data, sends the packet out over the network, and measures the round-trip elapsed time. The Python programming language was used to implement the benchmark program. Python is an object-oriented programming language that compiles into byte-code and runs on a virtual machine [similar to Java, .NET, Perl, and others]. It was selected because:

1. Python is safe, readable, and easy to modify (for trial and error experimentation)
2. The Python implementation does not require the license of a compiler or development license, making it convenient for any third party to recreate the program for independent execution of the benchmark test
3. The win32all extensions provide direct access to win32 system calls used to exercise serial ports.

In order to run the benchmark program (shown in Appendix A), a Python implementation will have to be installed on the host system. Installation of Python is described below (for more information on Python, visit http://www.python.org/)

Installing Python. Python installs from http://activestate.com/. The package contains the standard Python implementation plus some additional tools and libraries (including the win32all library that is used by the benchmark program). The download/installation process is as follows:

1. Download ActivePython from http://www.activestate.com/Products/Language_Distributions/
2. Double-click on the downloaded file (e.g. ActivePython-2.2.2-224-win32-ix86.msi) to install ActiveState Python. Select "typical" installation (requires approx 35MB of disk)

The "stock" version of Python can also be downloaded by visiting http://www.python.org/2.2.2/. The win32all extensions must then be separately downloaded from the following site: http://starship.python.net/crew/mhammond/win32/Downloads.html. The default location for Python download/installation differs from that used by ActiveState Python. As a result, the path of the Python executable in the DOS batch file used to run the benchmark program must be modified before the benchmark program can be run.

Once the benchmark file is acquired, the unzipped file then creates the following two files: bench.py (the Python program used to measure round-trip latency on a COM port) and bench.bat (a DOS batch file used to run bench.py). If the Python executable file is installed in a location other than C:/Python22/python.exe, the path will have to be modified accordingly. The benchmark program is run from a command prompt in the directory where bench.bat and bench.py are located: C:\> bench.bat COM1. This command will write single byte data blocks to COM1 and measure the time that elapses until the data is read back. The measurement will be repeated for 100 iterations with a delay of 100ms between iterations. Finally, the min, mean, max, and standard deviation of the measured delays (in milliseconds) will be displayed.

The following options may be specified on the command line (before the COM port parameter):

In this example, the baud rate is set to 115200 and the number of iterations to 10. The theoretical minimum time for writing and then reading a single byte at 115,200 baud, using a 16550 UART (with FIFO enabled), is approximately 0.3ms. It appears that the normal system overhead (including the benchmark program and system calls) is less than 0.3ms. This overhead will be constant from one test to another and is quite small compared to the latencies we are going to measure over the network.

The benchmark test was executed using the following parameters:

 The benchmark test was executed using the following parameters:

Serial Shoot-out: Products Tested

* Information taken from the CDW (catalog) Web site (http://www.cdw.com/) on 2/7/03 with the exception of the Digi One Realport IA, which was obtained from B&B Electronics (http://www.bb-elec.com/ on 2/07/03).

The Serial Shoot-Out measures the performance of leading serial device servers against each other and against native PC-based serial ports. It is not feasible to test every product, so a representative sample of comparable product was selected. Selection criteria included price, configuration, 2002 market share (NPD Intellect adjusted), and a serial COM port driver.

BENCHMARK RESULTS

An examination of the results obtained from the benchmark testing reveals a wide range in performance among the tested products. Latency is represented as the elapsed time (in milliseconds) required to execute one iteration of the network communications test. Therefore, the LOWER NUMBERS REPRESENT SUPERIOR PERFORMANCE.

To establish a common performance baseline, the test first was performed using the internal serial port of the Windows 2000 host PC. This port, COM 1, registered an average time of 5.35 ms to complete the loopback test at a port setting of 9600 bps. By establishing this number for a standard 16550 UART-based COM port, a reasonable conclusion can be reached that similar results achieved by a network-based COM port indicate performance rivaling an in-server multiport card or native PC COM port. The table below lists the model information and results obtained for each product included in the benchmark test.

Please contact Comtrol to obtain the test result from the independent test service retained by Comtrol to separately execute this benchmark.

COMTROL vs. DIGI - BENCHMARK COMPARISON

In October 2002 Digi International issued a press release extolling their Digi One IA RealPort product's latency ranking in a comparison of competing serial port (device) servers. The latency benchmark was supplied by Digi, and performed by VeriTest on Digi's behalf. The VeriTest report compared the latency of several serial port (device) servers from various manufacturers in a test that sent one data byte from a "master" PC via serial communication to a device server and on to a "slave" PC. This test measured the elapsed time for that data to make the round trip from the "master" PC, through the device server to the "slave" PC and return to the "master" PC. The test results proclaimed that the Digi IA RealPort bested the field of competitors, which included Comtrol, Lantronix, and Moxa, by what VeriTest categorized as "significantly lower round-trip latencies". The following is a direct reprint of the Digi press release announcing their benchmark results as published on October 16^th, 2002.

VeriTest Report Confirms That Digi One IA RealPort Performs Significantly Better Than Competitors(Minnetonka, Minn., October 16, 2002) - Digi International®, Inc. (Nasdaq: DGII), the leader in Connectware, today released a report summarizing the results of latency performance tests which showed that when tested against competitors including Lantronix, Moxa and Comtrol, the Digi One IA RealPort™ generated the lowest latency of all device servers tested.Key findings of the report stated that: "The Digi One IA RealPort device server generated significantly lower overall average latency compared to the other device servers we tested." The performance tests were conducted by VeriTest, a leading provider of independent outsourced testing solutions, and a division of Lionbridge Technologies, Inc. (Nasdaq: LIOX).Many industrial and control applications require low latency in order to operate reliably, with maximum throughput. These include discrete control systems, CNC/DNC and critical event reporting. Often, these systems were based on the performance of direct wire line connections, without assuming the overhead of crossing network boundaries. "The lower the latency generated, the better the speed and throughput for time critical applications," explains Joel Young, vice president of device server products for Digi International. "These test results clearly demonstrate that Digi products provide the fastest way to and through your network, and prove without a doubt that Digi is the leading provider of state-of-the-art device server products."The test measured the average latency over 10,000 iterations, and consisted of a master application sending a single 8-bit character to the device server under test and then waiting for the character to be received by the slave application and echoed back to the master. Latency results were measured in milliseconds and the results were as follows:Digi One IA RealPort: 5.81

Digi Press Release ContinuedThe Digi One IA RealPort is part of the Digi One™ product suite, which is designed to network enable virtually any serial device in the industrial automation environment. The product allows connection of new and legacy serial devices and PCs from anywhere on the factory floor, using any operating system. Advanced Modbus, Ethernet/IP, Allen-Bradley Ethernet, DF1, Compoway/F, FINS and Hostlink features allow for improved sharing of information between various factory floor devices, and advanced SNMP features allow for management of the industrial device from a centralized location. These features make the Digi One IA RealPort more versatile and cost-effective than other products on the market.Joe Dunsmore, president and CEO of Digi International, said, "These tests further validate the quality of our device server products, and once again verify our position as the leading provider of innovative device networking solutions. We remain committed to providing our customers with the products that will make their daily operations easier and help their businesses grow."

Comtrol Review of Digi's Benchmark White Paper

Both Comtrol and Digi "white papers" agree on the cause and effect that latency has on the feasibility of implementing serial device servers. But when the results of the testing are compared, Comtrol and Digi differ significantly in one area - Digi's performance!

Three points should be noted. First, based on Comtrol's benchmark, Digi's claim that its Digi One IA RealPort has " …  significantly lower overall average latency compared to the other device servers we tested" would be incorrect. Second, in Digi marketing information, the results of this benchmark are generalized to sound like all Digi serial device server products achieved this level of performance. It is clear in reviewing the benchmark that in each case (except the Lantronix UDS10 and UDS 100) different products from the same vendor produced different latency. The Digi benchmark (report) showed test reports for only one Digi product. Finally, Digi has not made its test program available nor do they offer an explanation for the rationale of using an unusual network topology (placing a "slave PC" into the configuration) when in practice, the data path is clearly from the host PC to the serial device server and directly back to the host PC.

With the exception of the Digi One IA RealPort, most of the other products tested by Digi and Comtrol fall within a reasonable tolerance. The improvement in Comtrol product performance between the Digi test and the Comtrol test can be explained quite easily. Comtrol offers a standard configuration option in all serial device server drivers that maximizes network performance in situations where latency is an issue. In a benchmark such as this one, this well-documented feature should have been activated by Digi - and if it had been, Comtrol's superior performance would have been clearly demonstrated!

Since the results achieved by Comtrol were so dramatically different, the test was re-run on the Digi IA RealPort multiple times - with nearly the same latency observed in each instance. Meanwhile, the latency of the other non-Digi products, including Comtrol products, approached the latency reported by VeriTest/Digi. Question: How can all the competing products achieve nearly the same latency and the Digi latency be 18 TIMES SLOWER in the Comtrol test?