Vulnerabilities in Operating-System Patch Distribution

Matt Power
BindView Corporation, RAZOR Team
mhpower@bos.bindview.com
18 December 2000
 

Coverage

In this research project, BindView Corporation has studied the processes by which 27 operating-system vendors distribute security patches. The report focuses on vulnerabilities in these processes, with the hope that customers can use the information to assess the adequacy of the processes used by their own vendors, in both an absolute and comparative sense. Customers may wish to work with their vendors to identify any changes in these processes that may be warranted. The vendors included in this report were selected because we think each one produces an operating system that is regularly used on a production basis in commercial environments, and because at least one of the following was true 25 of the 27 vendors were contacted by e-mail on 8 November 2000 with a draft copy of the full text of the section of the report that discussed that vendor's operating system. The e-mail was sent to all of these vendors within a 10-minute period on that day. The e-mail to each vendor included questions about some of the details of that vendor's patch-distribution mechanism. The responses to those questions are incorporated into the text below. A bookkeeping error resulted in Cobalt being contacted on November 14, and IBM being contacted on November 20. (We received a reply from IBM, but did not receive a reply from Cobalt.)

There are several other operating-system vendors that could have been included, but were not. For example, there are dozens of Linux distributions that are either relatively unpopular, or that are rarely discussed in English language forums. There are at least a dozen other vendors of operating systems (for general-purpose multi-user computers) that are not based on Linux. We are willing to extend this report with information from any other major vendor who would like to send it to us. (By major vendor, we mean one whose operating-system software is in use at at least a few thousand commercial sites. However, we will try to add information from smaller vendors as time allows.)

It is currently common practice among vendors to distribute all security patches for free over the Internet, and to allow these patches to be accessed by anyone, regardless of whether the person can prove he is a customer. Some vendors require that one purchase a support contract in order to obtain the full set of patches including the ones that are not related to security. This report primarily discusses free Internet distribution methods; in some cases, this is not the approach that the vendor recommends for contract customers to obtain patches.

Authentication methods and definitions

This report primarily addresses the issue of how cryptographic methods, such as digital signatures, are used for authentication (and the accompanying integrity protection) of patch delivery. We found that three methods are in use for providing cryptographic authentication of operating-system patches. First, PGP may be used -- and this can occur in a number of different ways: detached signatures, signatures that are part of documents, and signatures that are included as a specified field in a package file. (Often, GnuPG is used instead to obtain functionality similar to what PGP provides. For simplicity, we will refer to "PGP" when we mean either PGP or GnuPG.) Second, it may be possible to download patches over an https connection. Third, it may be possible to download patches over an ssh connection. PGP signatures are currently the most common, but this report is not recommending that PGP is the single preferred method. The "Differences in authentication methods" section below expands on this.

By "authentication", we mean that the patch is accompanied by information about the identity of a specific entity who affirms the validity of the patch, and we mean that the patch was not modified in transit. In other words, in this report, the word "authentication" should be taken to imply that there is also integrity protection. (Conversely, the term "integrity protection" would not imply authentication.) By "delivery", we mean any mechanism by which a patch that is distributed by an operating-system vendor reaches a system administrator who wishes to install the patch. This includes mechanisms that have an active component (e.g., the vendor sends e-mail to its customers announcing the existence of a patch) and ones that are completely passive (e.g., the vendor places the patch on its web or ftp server and expects that users will look for it there).

Threat models

Authentication of patches is valuable in addressing a wide variety of threats that can cause a user who thinks he is obtaining a correct patch from an authorized distribution site to actually be obtaining a trojan-horse patch from the attacker. The threats of interest in this report exist because of one of these deficiencies in the signature process: Some of the many other threats that we regard as significant, but that we are (for the most part) intentionally not including in the scope of this report, are The threats of interest are the specific techniques that an attacker might use that could result in a trojan-horse patch being obtained by a user who is actively trying to obtain a correct version of the patch. We expect that all of these threats have been previously identified by others, although some of these threats have probably not yet been successfully operationalized in the wild. In other words, we are not attempting to offer a novel threat analysis, but simply enumerating known threats in order to provide motivation for patch-authentication concerns. Many of the threats involve active participation by a host under the control of the attacker. In some cases, it would in theory be straightforward to track down the attacker's location and use that information to stop the attack. Presumably a clever attacker would use a compromised host that he controlled remotely via a hard-to-trace mechanism. Depending on a number of factors (such as lack of expertise at the local ISP responsible for that compromised host), it may be possible for the attacker to successfully distribute a large number of trojan-horse patches before the compromised host is shut down. Our enumeration of the threats includes: There is also the question of what type of a trojan-horse patch (or software update) would be distributed. Specific trojan-horse distributions that have been made in the past include ones for One limitation of these previous incidents is that the nature of the software change was not sufficiently subtle, leading to the change being noticed fairly quickly (also, information about the trojan horse's existence was distributed very widely). First, it would probably be more effective for the attacker to introduce a change in a compiled binary that was part of a software update, rather than introduce a change in a source distribution. Second, detection might be avoided for longer if the trojan-horse code only sent information out from the compromised machine, rather than provided a way for the attacker to obtain an interactive shell. Third, it would likely be in the attacker's interest to encrypt any information sent out from the compromised host. Fourth, to help avoid detection, the information sent out should use a protocol that would be already in frequent use on the victim machine (e.g., the information might be included in header lines of http GET requests). Fifth, detection chances would probably be reduced if the rate of outgoing information were kept small. We believe that all of these guidelines for trying to ensure trojan-horse survival have been identified previously by others. We are mentioning them primarily in anticipation of responses saying that "trojan-horse patches would always be detected promptly and nearly all affected users would be alerted". Also, it may easily be the case that hard-to-detect trojan-horse software updates have already been distributed but no one has yet discovered this (and, admittedly, this might never have happened yet).

Authentication of files that are not patches

Many of these same issues are threats affecting the online distribution of other files, such as the original operating-system distribution and optional packages that can be installed later. Authentication of the original operating-system distribution is of most interest when this distribution contains some PGP signed files, it is downloaded over the network (rather than obtained from a CDROM), and the distribution files are first downloaded to an already installed trusted system. Creating CDROMs with trojan-horse operating-system distributions and mailing them to people who own one's intended victim machines is certainly a possible attack method, but is outside the scope of this report.

If one installs an operating system directly over the Internet from an external distribution server, running a program such as PGP on the newly installed system (in order to verify the newly installed system's integrity) is of limited value for authentication. If an attacker were able to place a trojan horse onto the newly installed system, the attacker could also have arranged for the PGP program to report that all of the signed files have correct signatures from the expected userids. (This could conceivably involve a change to the PGP program itself, or a change to any other operating-system component that might be involved in running PGP, e.g., the base kernel, kernel modules, shared libraries, a shell, etc.) To make this signature spoofing very difficult to detect may require a very large amount of effort on the part of the attacker. Still, in a case where the attacker can control both the signed data and the signature-checking software, the attacker has the advantage.

To take the advantage away from the attacker would involve downloading the operating-system distribution onto a trusted machine of one's own, checking the PGP signatures of all of the downloaded files, setting up this machine as an installation server, setting up a trusted network connection between this machine and the new machine, and then installing the new machine. This is somewhat cumbersome, and substantially increases installation time in cases where there is only one new machine to install. The issue of how to obtain the first trusted machine is also significant. There is also the problem that, currently, few if any operating-system vendors provide a PGP signature for every file in the distribution. For example, some Linux vendors provide a PGP signature for every package but do not provide a PGP signature for the downloadable boot-floppy image. Also, BSD Unix vendors typically provide some files that contain MD5 checksums of the operating-system distribution files, but the checksum file is not PGP signed. (This might change, in some cases, in the future; for example, see the FreeBSD section below).

In general, in cases where the installer has the resources to use this cumbersome installation procedure, the PGP signatures on the operating-system distribution files are of significant authentication value. However, we regard PGP signatures on patch files as more significant. Some patches need to be applied on machines that are already in production use. Thus, if a trojan-horse file has a malicious effect that occurs immediately and is noticed immediately, the consequences can be much greater if it is a patch than if it is part of the original operating-system distribution. Also, it is rarely necessary to obtain the original operating-system distribution over the network. One can instead obtain it via CDROM, which is safer at least in that the range of possible attacks is smaller. Obtaining patches via CDROM is sometimes possible, but may be completely infeasible if a critical security patch needs to be obtained and installed immediately.

The availability of digital signatures on packages (or, generally, new software components added after installation time that are not intended to correct problems with existing software) is useful, but again not quite as critical as a digital signature on a patch. This is because it is seldom as urgent to install a new package as to install a patch, and thus the package might be obtained on a CDROM rather than via the network. The two may be equivalent in the case of some operating systems (such as most Linux systems) in which all patches are distributed in the form of a package. Here, a single updated package file may be installed by some customers as a patch that supersedes a vulnerable package, and installed by other customers as a newly added software component.

Differences in authentication methods

In each of the authentication methods (PGP, https, and ssh), there are some requirements on the user's behavior for the authentication to be useful. In most cases where PGP is used, the user needs to manually check that a valid signature was made using a userid that is supposed to be associated with the vendor's patch-distribution process. When https is used, the user needs to manually check that the server certificate associated with the https connection is one that belongs to the vendor (e.g., "View", "Page Info" in Netscape's browsers; "File", "Properties", "Certificates" in Microsoft's Internet Explorer). When ssh is used, the user needs to ensure that the ssh public key obtained from the server matches the one that the vendor installed on that server. (In practice, this would typically be done by obtaining a copy of the ssh public key either in a PGP-signed format, or via https. As long as the ssh key is checked manually upon one's first access to the ssh server, it is typically safe to allow the ssh software to automatically verify that the key is unchanged upon each future access to that ssh server.)

In the discussion of the individual vendors' processes, we are envisioning a security-conscious user who actually goes though these manual verification steps. (This user would also know to avoid engaging in https sessions to sites that do not have server certificates signed by a recognized certification authority, or that have an expired server certificate.) We are aware that readers will comment on this report by, for example, dismissing all of the https authentication as irrelevant because ordinary users rarely, if ever, check the certificate details for their https connections. Our point is that https authentication is useful, to some extent, in that users can check the certificate details. Checking the certificate details is supported by major web browsers, it is not especially cumbersome, and the certificate details are presented in a way that is understandable even by those who are not experts on the X.509 and SSL specifications. Admittedly, if one is responsible for downloading many different patches from several different vendors' https sites in a limited amount of time, the additional workload needed for checking certificate details may be prohibitive.

There are other approaches to patch authentication that would not require the time-consuming manual steps. One is the SFS read-only file system (SFSRO), described at http://www.pdos.lcs.mit.edu/papers/sfsro.html. (This is also referenced in the FreeBSD section below, although the software is not part of FreeBSD nor is FreeBSD required to use it.) SFSRO also solves another important problem that detracts from the viability of vendor https sites. In https, the server incurs a large CPU load due to the continual cryptographic computations. One consequence is that any patch-distribution site that will start offering https may need a substantial hardware upgrade (which could involve replacing general-purpose servers, or adding specialized SSL hardware). In SFSRO, the bulk of the cryptographic computations for a certain set of content occurs only once, regardless of how many files are downloaded or how many clients connect to the server. (There is a small per-connection load.) Also, these computations do not need to be repeated even if the content is distributed from multiple mirror sites. Each client has cryptographic computations to perform, but that generally does not result in performance degradation or noticeable delays on typical client hardware.

A third common drawback of many https applications is one that is largely inapplicable to patch distribution. There are many (probably thousands of) small online shopping sites that allow payment details to be entered via https, but the https session is associated with a certificate that does not belong to the site's owner. For example, the certificate might be owned by "third-party-credit-card-thingie.com". Thus, even if the user is careful to check the certificate details, they still have to decide whether they believe "third-party-credit-card-thingie.com" is legitimately authenticating https sessions used for payment to "itty-bitty-online-shopping.com". For patch distribution, the problem does not occur because the operating-system vendors all either own their own server certificate, or do not use https at all.

A fourth drawback of https is that it may be considerably less effective for authentication in cases of mirror sites. If there is an https mirror site of a particular free operating system's patch collection, this does not imply that the site operator obtained those patches from a "master" site via an authenticated connection. It may very likely mean that the web server housing the mirror site has https enabled for an unrelated reason, and the availability of the patches via https is simply an unintended side effect. In general, the authentication value of a particular copy of a patch depends mostly on the authentication value of the most weakly authenticated step involved in establishing that copy. Also, in the case of free operating systems, it may not be economically feasible for all mirror sites to have server certificates issued by a recognized certification authority. In many cases, the mirror sites are operated by volunteers, and the management of the free operating-system project may not have any control over the host security or operational practices of the mirror site. Thus, it would not necessarily be appropriate for the mirror sites to have server certificates that were issued with the name of the free operating-system project as the subject name.

A fifth drawback of https is that, if an attacker is able to cause his own content to be served from a vendor's web site, that rogue content will have the same apparent authenticity as the vendor's intended content. Rogue content can consist of content that the attacker actually uploaded to the vendor's server, or (in some cases) content elsewhere if the vendor's server acts as a proxy server or acts as a client of a distributed file system. Because of this, it might be thought that the PGP approach is better, in that explicit action is needed to authenticate any file. However, this is not necessarily true. One could imagine that a non-https web server might dynamically generate detached PGP signatures whenever a filename ending with ".asc" is requested. Even if signatures are not dynamically generated, the PGP private key file might have an empty passphrase and be stored on the web server in an easily located directory, such that compromising the server essentially implies compromising the private key. For most vendors that use PGP, there is no documentation on their web site explicitly stating that the private key is protected any better than the patch-distribution server. (There are some exceptions, e.g., a Debian web page mentions "Be very careful with your private keys. Do not place them on any public servers or multiuser machines, ...."; and there are key-security plans mentioned in the Immunix section below.)

It seems likely that many vendors keep their PGP private key off of their patch-distribution server, and that some vendors keep their PGP private key on a machine that has no network connection. (Similarly, with SFSRO it is recommended that content signing occur on a trusted offline machine; the content can then be copied onto any number of untrusted distribution sites.) Also, it seems likely that many vendors perform PGP signing on machines that are not at high risk of compromise (e.g., the signing machine is behind a usefully functioning firewall, and the signing machine is not used for e-mail or web browsing in ways that might introduce malicious software onto that machine). In these cases, PGP authentication may be considerably more trustworthy than https authentication. However, with the details of PGP key-handling procedures unknown, one cannot conclude that https authentication is necessarily inferior.

Some indirect information about the percentage of users who bother to verify PGP signatures of security patches (or patch announcements) can be obtained by counting the number of times that a vendor's PGP key has been downloaded. This is imperfect in that a user may have obtained a copy of the PGP key from a vendor's web site, from a PGP key server, or (in some cases) from a CDROM issued by the vendor. Also, a user who has downloaded the PGP key may not actually use it to verify signatures regularly (if at all). Still, we were interested in a report about the number of downloads of the file

http://www.microsoft.com/technet/security/MSRC.asc
A web page that links to this file (specifically, http://www.microsoft.com/technet/security/notify.asp) is mentioned in every Microsoft security bulletin as the location of the PGP key used for signing the bulletin. The Microsoft Security Response Center told us "I don't have precise numbers, but no more than several thousand people have downloaded our PGP key." Given that Microsoft has millions of customers, it would seem likely that only a very small fraction of affected customers are checking the PGP signatures on these bulletins.

Absence of cryptographic authentication

Some vendors do not provide cryptographic authentication of patches via PGP, https, ssh, or any other method. Presumably some readers are at organizations that pay large amounts of money to a vendor for product support, perhaps including expedited preparation of patches and private announcements of patch availability. If any of these readers are concerned about inadequacy of patch authentication, it would likely be a good idea to contact the vendor and request use of an authentication method that would be convenient for you. As a practical matter, vendors are much more likely to support PGP or https than ssh. It is possible that some vendors provide authenticated patch distribution to customers who have a support contract, but offer only unauthenticated patch distribution from their public Internet site. However, we did not find any vendors that advertised such an offering, or that mentioned it to us.

In general, it seems likely that the overriding reason for lack of patch authentication by some vendors is that there has been little or no known customer demand. Setting up patch authentication via some of the potentially desirable methods can have substantial costs (e.g., costs of more powerful hardware if the distribution service were completely changed from http to https, or costs for setting up a secure offline facility used for PGP signing). It is likely that all vendors are aware that most of their customers do not obtain or apply patches at all. The number of customers who both obtain patches and are dissatisfied with patch authentication may be very small. If your organization has enough leverage with a vendor to obtain changes in a patch-authentication process, you still may need to be realistic in what you request. For a number of vendors, a requirement that "all PGP signing must occur offline in underground vaults" would not be realistic. It might, for example, be the case that "PGP signing must occur a dedicated machine that is not Internet facing" is the strictest requirement that a particular vendor would be willing to adopt.

The cryptographic authentication methods are not the only possible methods of authentication. Some vendors supply an unsigned list of patch filenames and the corresponding checksums. In some cases, it is possible to download a patch from one distribution server operated by the vendor, and download the checksum file from a different distribution server that is owned by the same vendor but independently maintained. The existence of unsigned checksum files is not, however, a strong defense against a determined attacker. (The attacker would arrange for the user to obtain a modified version of the checksum file that contained the correct checksum for a trojan-horse patch.) In general, unsigned checksum files are valuable in determining whether a downloaded file was accidentally corrupted. They do not provide a substantial security benefit, and in the individual-vendor sections below, we have omitted mention of whether a vendor supplies any unsigned checksum files.

Although a number of attacks are possible against the patch-delivery process when it involves unsigned files obtained via http or ftp, these approaches are still significantly better than receiving unsigned patch files via e-mail. Although we do not know of any cases in which vendors advertise that they will sometimes send unsigned patch files to their customers via e-mail, we would recommend against this approach. The limited trust that one can associate with http and ftp still exceeds the trust that one can associate with e-mail, especially if the vendor were to originate the patch e-mail from source IP addresses that were not previously announced.

Scope extension

Although this report focuses on the vendors' apparent authentication capability, we also include some limited analysis of host security on the vendors' patch-distribution sites. The motivation for this extension to the scope was that we visited many vendors' ftp sites using an ftp client that displayed the "220" banner message, e.g., 
  220 ftp.some-os-vendor.com FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready.
and noted that several of them were running software that apparently predated the summer 2000 vulnerability findings described at
https://www.cert.org/advisories/CA-2000-13.html
(a format-string issue that can lead to remote root compromise). This apparent ftpd vulnerability was reported to each affected vendor as part of the original mail messages that were sent on 8 November 2000.

Again, some readers are likely associated with organizations that pay a vendor for product support, and this support may include the availability of additional patches located on the same patch-distribution site that the vendor uses for security patches. If any of these readers are concerned about host security on the patch-distribution site, it would likely be a good idea to request that the vendor arrange for an independent security audit of that site. (Please note that BindView Corporation does not provide such security-audit services, and has no current plans to expand into that business area.)

The reader should not infer that we believe ftpd vulnerabilities are the primary mechanism through which patch-distribution sites might be compromised. The mention of possible ftpd vulnerabilities at many vendor sites was included only because the ftpd version number was, figuratively, staring us in the face as we attempted to access the ftp site for our other research work. We do, however, feel that ftpd vulnerabilities would be an especially significant threat to these sites, in that the patch-distribution process requires that the ftp port be open and accessible to the Internet; other common vulnerabilities might be absent if host security practices resulted in other network daemons not running, and other ports blocked by a firewall. In general, our observations about ftpd vulnerabilities at some major vendor sites suggest that these sites are not well defended from breakins, and thus PGP signatures made on a trusted offline machine would add considerable value.

Discussion of Individual Vendors

In the sections about individual operating-system vendors, we are expecting that readers of each section already have at least a basic familiarity with a vendor's product offerings. For example, we do not specifically mention that the NetBSD base operating system is freely distributable, whereas the Windows 2000 base operating system is not. The vendors who supply free operating systems do not charge a fee for subscriptions to their mailing lists that announce security patches, so we did not specifically mention that in each section. When vendors are listed, sometimes the format "Vendor-Name / OS-Name" is used for clarity. This does not imply that that format is regularly used (i.e., the specific wording "Caldera / OpenLinux" is usually not used when referring to the OpenLinux operating system).

Apple

There have not been many security-related patches issued for Apple products. One security-related update was issued in January 2000; it was a new version of Open Transport that "prevents Macintosh computers from being used in certain types of Denial of Service (DoS) issues." The update was provided via the web page http://asu.info.apple.com/swupdates.nsf/artnum/n11560. The update is apparently not accompanied by a PGP signature or other type of digital signature. Also, the update is apparently not available via https, and thus customers cannot use the authentication provided by a server certificate to help ensure that they are obtaining an unaltered patch issued by Apple.

Some older patches or updates for the NeXTSTEP operating system are obtainable from ftp://ftp.next.com/, which is located on Apple's network (IP address 17.254.3.217). The banner provided by the ftp.next.com server is 

  220 enterprise FTP server (Version wu-2.4(3) Tue Jul 8 01:56:55 PDT 1997) ready.
This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of
https://www.cert.org/advisories/CA-2000-13.html
We sent the server a test "site exec" string to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet ftp.next.com 21
  220 enterprise FTP server (Version wu-2.4(3) Tue Jul 8 01:56:55 PDT 1997) ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %08x%08x%08x%08x%08x%08x%08x%08x  (with several more %08x)
  200-000181c0bfff00c5bffffd6e  (actual response contained additional data)
  200  (end of '%08x%08x%08x%08x%08x%08x%08x%08x')  (continued with more %08x)
where 000181c0, bfff00c5, and bffffd6e were apparently successive values in the ftpd process memory. This server does not seem to support a format-string syntax that would allow commands such as "SITE EXEC %12345$08x" to read individual values from process memory at specified locations. This property, along with limitations on the length of command lines that the ftpd will read, may limit any vulnerability caused by format-string issues.

BSDI / BSD/OS

BSD/OS security patches are listed on the web page https://www.bsdi.com/services/support/patches/, with a certificate from the Verisign/RSA Secure Server certification authority used in the authentication process. There are also detached PGP signatures for the patches, made using the signing key "Berkeley Software Design, Inc. <bsdi@BSDI.COM>." The PGP key can be obtained from https://www.bsdi.com/pgp_key, again with https authentication via the Verisign/RSA certificate. This file includes a telephone number that can be used by customers wishing to verify this PGP key.

Caldera / OpenLinux

Security patches for OpenLinux are announced via mailing lists such as announce@lists.calderasystems.com, and on the web site http://www.calderasystems.com/support/security/. The announcements are always PGP signed using the "Caldera Systems Security <security@calderasystems.com>" signing key.

A security "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files themselves have embedded signatures using PGP, again using the security@calderasystems.com signing key.

The signatures on the security@calderasystems.com key appear to be sufficient for it to be well trusted in terms of the PGP trust model. Also, the key is included on the CDROM that contains the OpenLinux distribution.

Most OpenLinux packages are open source and this provides an opportunity for source-code review by the user.

Cobalt

Security patches to the software for Cobalt's products are announced via the cobalt-announce@list.cobalt.com mailing list, which anyone may join at no cost. The announcements do not contain PGP signatures or other types of cryptographic digital signatures. The patch files are distributed via http and via ftp from the server ftp.cobalt.com. A "patch" actually consists of a file with a name ending in ".pkg", which is in gzipped tar format. The .pkg file contains a few files used in installing the software update, as well as some implementation of an actual security-related change. The security-related change in some cases consists of a new .rpm file that contains software incorporating a security fix. In other cases, the security-related change consists of a patch (in diff format) that applies to text files that are part of Cobalt's original software distribution.

When a .rpm file is distributed, it may be a .rpm file packaged by Cobalt, and it may be a .rpm file packaged by Red Hat. Cobalt packages all .rpm files that are specific to the mips architecture that is used for some Cobalt hardware platforms. Cobalt also packages some .rpm files that are used on the i386 architecture, and some .rpm files that are architecture independent.

The Red Hat .rpm files distributed by Cobalt can be authenticated using the standard mechanisms for authenticating a Red Hat .rpm file, i.e., the file has an embedded PGP signature with the "security@redhat.com" signing key. The .rpm files packaged by Cobalt do not contain embedded PGP signatures. Also, except for the presence of Red Hat .rpm files in some subset of the .pkg files, no part of Cobalt's distribution of .pkg files is accompanied by a PGP signature or other type of cryptographic digital signatures. Also, the patches are apparently not available on an https site.

Most of the software for Cobalt's products is open source and this provides an opportunity for source-code review by the user.

Compaq / Tru64 UNIX

Security patches for Tru64 UNIX and other Compaq operating systems are announced via a mailing list named "security", which anyone may join at no cost by typing in their e-mail address at http://www.support.compaq.com/patches/mailing-list.shtml. The patches are also listed on the web site http://ftp.support.compaq.com/patches/.new/security.html. Additional information about the patch process can be found using the links on the page http://www.support.compaq.com/patches/.

Neither the patch files nor the announcements are accompanied by PGP signatures or other types of cryptographic digital signatures. Also, https authentication is apparently not available. Compaq does operate some https sites, including https://www.directpluspro.compaq.com/, which provides authentication via a certificate from the Verisign/RSA Secure Server certification authority. However, apparently neither the patch contents nor the patch descriptions can be obtained from an https site.

Compaq is working on a new process for the distribution of Tru64 UNIX and OpenVMS patches, but no details about this have yet been announced. Compaq has mentioned two approaches that they currently use to try to ensure the integrity of their patches. First, the servers that store patches are regularly scanned with antivirus software. Second, there is a separate server that is used for copying patches onto the publicly accessible patch distribution site. This separate server cannot be accessed directly from the Internet, and also is isolated from almost all of Compaq's corporate network (except for some other protected systems that require access to it).

Conectiva Linux

Security patches for Conectiva Linux are announced via the mailing list named atualizacoes-anuncio. This mailing list is available only in the language pt_BR. The same announcements are also available in both English and Spanish, but those versions are sent out on mailing lists that are not operated by Conectiva itself (e.g., one source of the English language version of Conectiva's announcements is the bugtraq@securityfocus.com list). The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files themselves have embedded signatures using PGP. The signing key used is "Conectiva S.A. <security@conectiva.com.br>". This key is available from https://www.conectiva.com.br/contato/pgp-gpg-conectiva.key.zip, with https authentication provided by a certificate issued by the Thawte Server certification authority. The key is also included on the CDROM that contains the Conectiva Linux distribution.

Most Conectiva packages are open source and this provides an opportunity for source-code review by the user.

Corel LINUX OS

A Corel LINUX OS release includes a number of Debian/GNU Linux packages, and a number of other packages that are either specific to Corel LINUX OS or are modified for use in Corel LINUX OS. There is currently one security patch announced for a piece of software that is not a Debian/GNU Linux package, specifically, the Corel Update application. There is a link to the announcement from http://linux.corel.com/support/updates.htm. The announcement is not accompanied by a PGP signature or other type of cryptographic digital signature. Also, the announcement is apparently not available on an https site.

The link to the patch itself uses a URL that lacks a domain name

ftp://ftp/pub/linux/CorelLinux/dists/old/corel/binary-i386/
and there is no patch for Corel Update available at
ftp://ftp.corel.com/pub/linux/CorelLinux/dists/old/corel/binary-i386/
Still, http://linux.corel.com/support/updates.htm is likely a good web page for Corel customers to check if they are interested in future patch availability (keeping in mind that there may be significant changes if Corel's Linux business is sold to another company).

The Debian packages that are included in a Corel LINUX OS release can be authenticated using the standard mechanisms for authenticating a Debian package, i.e., there is a PGP signature in a ".dsc" file associated with the package. The packages specific to Corel LINUX OS appear to have unsigned .dsc files, e.g.,

ftp://ftp.corel.com/pub/linux/CorelLinux/source/corellinux-1.2/corel/shadow_980403-cl-1.0.dsc
suggesting that authentication via PGP signatures or other types of cryptographic digital signatures is not available. However, persons concerned about authentication of the online distribution can refer to the copies of these files located on the CDROM distribution. This does not address all distribution security concerns but does eliminate some possible mechanisms for trojan-horse distributions that are specific to the network distribution method.

Most Corel LINUX OS packages are open source and this provides an opportunity for source-code review by the user.

Debian GNU/Linux

Security patches for Debian GNU/Linux are announced via the debian-security-announce mailing list, and include MD5 checksums of the new files available for download. (The "patch" actually consists of a new package file known as a ".deb" file, accompanied by ".orig.tar.gz", ".diff.gz", and ".dsc" files). All of the announcements are PGP signed by one of four persons; their keys are available at http://www.debian.org/security/keys.txt. For two of the PGP keys used, the signatures on the key appear to be sufficient for it to be well trusted in terms of the PGP trust model. The two other PGP keys used are, apparently, each signed only by the key creator himself or by one other person, and these two keys do not seem well trusted in terms of the PGP trust model (this does not imply that the keys are inappropriate for debian-security-announce messages).

The information provided by the www.debian.org web site and the services on db.debian.org does not appear to provide additional cryptographic authentication of the validity of the four keys. For example, http://www.debian.org/security/ includes a statement about the PGP keys of the security team. However, Debian's servers apparently do not have a similar statement available via an https service that uses a certificate from a recognized external certification authority. Also, Debian's servers apparently do not include similar statements from other participants in the Debian project, signed with PGP. (The security team keys themselves are available via https://db.debian.org/ which has a self-signed certificate.)

The file

ftp://ftp.us.debian.org/debian/dists/woody/contrib/source/misc/debian-keyring_2000.08.30.dsc
is signed by "James Troup <james@nocrew.org>". However, our understanding is that this signature does not constitute a statement by James Troup that he is a central signing authority who is asserting that each of the keys in the keyring belongs to the person named in the key's user ID. It is only intended as a statement that James Troup used the indicated version of the debian-keyring .tar.gz file when building the package.

The organization of the Debian project limits the extent to which a central signing authority could be implemented in a viable and meaningful way. A central signing authority could, in some cases, make it more difficult for an attacker to successfully get a Debian user to install a trojan-horse package. However, realistically a central signing authority would not have personally met every Debian package maintainer and thus could not provide a first-hand statement about whether that person's key is absolutely valid for inclusion in debian-keyring. A central signing authority could act on the basis of indirect knowledge about the individual keys, but authentication provided that way might lead to a false sense of security (i.e., users might then trust the overall security of package signing more than they actually ought to trust it). In general, it appears that the digital-signature practices that are in place are reasonable for the current structure of the Debian project. Also, even though a corporation may issue a Linux distribution with every package signed with a single key such as "security@well-known-linux-company.com", it would almost certainly not make sense for Debian to do that.

Many Debian package maintainers participate in organized key signings in an effort to make more of the keys in debian-keyring ones that would be considered well trusted in terms of the PGP trust model. This should lead to growth in the value of the authentication available for Debian packages. Also, when a new person wishes to become a Debian developer, it is recommended that they meet at least one already active Debian developer in person (i.e., not an online meeting). These meetings (and other procedures used when an in-person meeting is not feasible) are used to obtain identity and key-parameter information that is needed for one person to agree to sign another person's PGP key.

The element of a distributed Debian package that includes the digital signature is the ".dsc" file. (There is also a ".changes" file that is signed, but the role of this file is in Debian's process for making new files uploaded by developers become available for distribution.) The .dsc file includes the MD5 checksums of the complete source code used for building the package -- this file is signed using PGP. (The .dsc file does not include any checksum of a binary file or collection of binary files.) As in the case of the announcement signature keys, the keys used for these .dsc signatures sometimes are ones that would be considered well trusted in term of the PGP trust model, and sometimes are not (this does not imply that some of the keys used are inappropriate for use in signing packages).

The announcements all relate to security fixes for free software; the source availability provides an opportunity for source-code review by the user.

FreeBSD

Security patches for FreeBSD are announced via mailing lists including FreeBSD-security-notifications@FreeBSD.org. The announcements are PGP signed using the FreeBSD Security Officer PGP key -- the signatures on this key appear to be sufficient for it to be well trusted in terms of the PGP trust model. The announcements in some cases include full patch source code. In other cases, there is a patch file distributed via ftp; however, the patch file is always accompanied by a detached PGP signature, made using the FreeBSD Security Officer PGP key. Also, sometimes there are auxiliary files related to the advisory, and the announcement will refer to the location of those files and include an MD5 checksum of their content within the PGP-signed text.

Patches consist of source code changes, and the base source code (ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/4.2-RELEASE/src/) is apparently not itself PGP signed. The source availability provides an opportunity for source-code review by the user. Also, persons concerned about authentication of the online distribution have the option of purchasing the base distribution (including its source code) on CDROM. This does not address all distribution security concerns but does eliminate some possible mechanisms for trojan-horse distributions that are specific to the network distribution method.

For future releases, FreeBSD will be looking at including PGP signatures for some files in the distribution such as the src/CHECKSUM.MD5 file. (There has been no announcement that this will definitely happen; they will consider making it happen, however.)

FreeBSD does not provide an authenticated mechanism for obtaining the current source code via CVS. Although some vendors of free operating systems provide source access via anonymous CVS over ssh, FreeBSD does not do this. In part, this is because CVS was not designed to prevent attacks that allow a user able to access a machine via CVS to also gain shell access to that machine. This issue was discussed extensively on the bugtraq@securityfocus.com mailing list in late July and early August 2000. For example, see

http://www.securityfocus.com/archive/1/72698
FreeBSD allows anonymous access to source-code repositories via CVSup, and although this could in theory be extended to provide authentication, in practice that will not occur soon because CVSup is written in Modula-3 and there are few developers interested in doing modifications to Modula-3 source code.

FreeBSD is considering whether other approaches to the authentication of its source code are feasible given FreeBSD's resources. FreeBSD will be evaluating whether SFSRO is a usable approach (SFSRO is a facility that associates a digital signature with an entire read-only filesystem in a way that provides users with authenticated access to any file; it is described at http://www.pdos.lcs.mit.edu/papers/sfsro.html). It is also possible that FreeBSD will create a public key infrastructure involving a FreeBSD Certificate Authority.

Hewlett-Packard / HP-UX

HP sends announcements of new HP-UX security patches to a mailing list called the Daily Security Bulletins Digest. Anyone may join this mailing list and there is no cost. Messages sent to this list do not contribute to authentication of the contents of the patches (e.g., via PGP-signed checksums -- the messages neither have checksums for the patch files nor are the messages PGP signed). Patch descriptions are available on the IT Resource Center "maintenance and support" http site. They are also available at a corresponding https site, and the http site has a link labeled "Secure Web server" to make it clear to customers that they can use https if they wish. Even if one lists the patches via the https site, though, the links for downloading the contents of the patches go only to http and ftp sites.

However, it is possible to download the patches via an https connection, with a certificate from the Verisign/RSA Secure Server certification authority used in the authentication process, if one is willing to manually enter a URL. For this to work, one apparently must register for a username and password using HP's European IT Resource Center web site (registration is free and does not require one to have an HP product serial number on hand). Then, one can download patches using URLs beginning with https://europe-ffs.external.hp.com/. The correct https URL can be found by examining what http://europe-ffs.external.hp.com URL is accessed by one's web browser during a download attempt at the European IT Resource Center's "Patch Database Shopping Cart", and substituting https for http. (Note that, despite the use of the term "Shopping Cart", there is no cost associated with patch downloads.)

The server certificate has the name europe-support2.external.hp.com; however, one apparently must instead use the name europe-ffs.external.hp.com to download patches (otherwise, the resulting web page will state that the patch is not available). A web browser may ask for user confirmation that one wishes to establish the https connection even though the server name was different than expected. (HP uses the names *-ffs.external.hp.com when referring to a host's role as a "Fulfillment Server", and uses the names *-support*.external.hp.com when referring to a host's association with HP's IT Resource Center.)

Note that, although downloading patches via https happens to work via this somewhat cumbersome approach, HP does not consider https patch download to be a service that it offers, and there are currently no plans to offer this service.

The patch files themselves at ftp://ftp.itrc.hp.com/ (and on web sites including europe-ffs.external.hp.com) are apparently not accompanied by PGP signatures or other types of cryptographic digital signatures.

There are also some patches or updates to a limited set of HP products available at ftp://ftp.cup.hp.com/. The help text provided by the ftp.cup.hp.com server is 

  214-The following SITE commands are recognized (* =>'s unimplemented).
     UMASK   CHMOD   GROUP   NEWER   INDEX   ALIAS   GROUPS 
     IDLE    HELP    GPASS   MINFO   EXEC    CDPATH 
  214 Direct comments to ftpadmin@ftp.cup.hp.com.
suggesting that it may have an implementation based in part on wu-ftpd.

This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of

https://www.cert.org/advisories/CA-2000-13.html
We sent the server a test "site exec" string to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet ftp.cup.hp.com 21
  220 onet4.external.hp.com FTP server (FTP) ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %08x%08x%08x%08x%08x%08x%08x%08x  (with several more %08x)
  200-0000000000000...343700003a323631000000003a315651  (some data omitted)
  200  (end of '%08x%08x%08x%08x%08x%08x%08x%08x')  (continued with more %08x)
where the 343700003a323631000000003a315651 data (split up appropriately) apparently corresponds to successive values in the ftpd process memory. This server does not seem to support a format-string syntax that would allow commands such as "SITE EXEC %12345$08x" to read individual values from process memory at specified locations. This property, along with limitations on the length of command lines that the ftpd will read, may limit any vulnerability caused by format-string issues.

IBM / AIX

IBM sends information about the availability of AIX security patches to multiple mailing lists. One of these is named security and is operated by aixserv@austin.ibm.com -- anyone may join this mailing list at no cost. Another mailing list is used for distribution of external releases by the IBM Emergency Response Service (ERS). A direct subscription to ERS notifications is available only as part of a for-fee service package; however, messages from the ERS are often posted to the bugtraq@securityfocus.com mailing list. The ERS external-release messages are PGP signed -- the signatures on the signing key used appear to be sufficient for it to be well trusted in terms of the PGP trust model. The signed messages include MD5 checksums of patch files, thereby providing authentication of the patch file contents.

In the past, messages sent to the security mailing list operated by aixserv@austin.ibm.com have not been PGP signed. However, after receiving BindView's inquiry about IBM's patch authentication, the AIX Security Team Lead stated that all future messages to that list will be PGP signed.

There are two types of AIX security patches that customers may see announced. First, for critical security problems, there may be an e-fix (emergency fix). The announcement of an e-fix is sent by the ERS, it is PGP signed, and the PGP-signed text includes an MD5 checksum of each associated binary file. Second, all security patches are eventually distributed in the form of an APAR. An APAR is an official IBM patch. (The term APAR stands for "Authorized Program Analysis Report"; however, the term is commonly used to refer to the patch itself, rather than used to refer to the problem report.)

There are two additional ways in which authentication of IBM patches can be achieved: use of https, and PGP signatures that accompany files on an FTP server. These are especially useful in cases where an IBM patch has been announced via a message without a PGP signature, or where the announcement does not include an MD5 checksum.

The https method is available for AIX patches -- they can be downloaded at URLs beginning with https://techsupport.services.ibm.com/aix/. This has a certificate from the Thawte Server certification authority used in the authentication process. However, since this web server apparently does not allow directory listings, one may need to use ftp://techsupport.services.ibm.com/aix/ to discover the file names of interest, and then visit the corresponding https URL to download the files over an authenticated SSL connection (the directory and file names are the same between the ftp and https servers, so the URL change is very simple).

(The server techsupport.services.ibm.com also has the name ftp.software.ibm.com and a few other names; we are using the name techsupport.services.ibm.com since that is the name on its https server certificate.)

Patches for other operating systems can be obtained via ftp from techsupport.services.ibm.com, but the directories containing these patches are apparently not available via https URLs (for example, the ftp://techsupport.services.ibm.com/ps/products/tcpip/fixes/v4.3os2/ic27721/ URL that is used for an OS/2 Warp ftp-server patch apparently does not have a corresponding https URL).

The FTP-distributed PGP signature method is in use on the server techsupport.services.ibm.com, which has some files with detached signatures signed with the security-alert@austin.ibm.com PGP key (again, an apparently well trusted key). An example is the directory /aix/efixes/security -- however, there are PGP signatures for only a small fraction of the patches contained in that directory. IBM plans to include PGP signatures for all new files uploaded to directories under ftp://techsupport.services.ibm.com/aix/ -- text files will include PGP signatures within the file, and binary files will have detached PGP signatures.

WireX / Immunix OS

Security patches for Immunix OS are announced via the bugtraq@securityfocus.com mailing list and perhaps via the stackguard mailing list. The announcements contain MD5 checksums of the new files to download, and are PGP signed using the signing key Greg Kroah-Hartman (Wirex Communications, Inc.) <greg@wirex.com>". The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files do not have embedded PGP signatures.

The greg@wirex.com signing key is self-signed and does not have any other signatures (either in the version available at http://immunix.org/~greg/greg_kh.asc or in the version we found via a PGP key server). This key is apparently not one that would be considered well trusted in terms of the PGP trust model. Also, apparently neither this key nor the Immunix OS packages are available via https.

These observations do not imply that the key is inappropriate for use in security announcements.

WireX plans to create a corporate PGP key soon, and will establish key-management practices to help ensure that the private-key portion is not compromised (e.g., keeping the private key off of Internet connected hosts and establishing strong physical security). Once that is done, WireX will publish the public key on its web site and arrange for it to become well trusted through such avenues as USENIX key signings. WireX will use this corporate PGP key for signing of future RPM package files that it distributes.

The Immunix OS version 6.2 packages are open source and this provides an opportunity for source-code review by the user. (Immunix System 7 is not entirely open source, and WireX will likely extend its production of non-open-source software in the future. There will continue to be open-source components, however.)

WireX plans to introduce a new trust model for patch distribution in the future; the details of this have not yet been released, however. They believe that their approach would be better than using https for distribution of patches.

MandrakeSoft / Linux-Mandrake

Security patches for the Linux-Mandrake operating system are announced via the security-announce@linux-mandrake.com mailing list. The announcements are PGP signed using the security@linux-mandrake.com PGP key, and include MD5 checksums for the patch files. The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm files have embedded signatures using PGP, again with the security@linux-mandrake.com signing key.

However, there appears to be no authentication of the validity of the security@linux-mandrake.com PGP key. The key is self-signed but has no other signatures (either in the version of the key stored at http://www.linux-mandrake.com/en/security/RPM-GPG-KEYS or in the version of the key obtainable from a PGP key server). Also, the key is apparently not available via https and thus the authentication provided by a server certificate is not available for confirming the validity of this key.

This does not imply that the key is inappropriate for use in security announcements and in package signing.

Most Linux-Mandrake packages are open source and this provides an opportunity for source-code review by the user.

Microsoft / Windows

Security patches for Windows operating systems are announced via the microsoft_security mailing list, which anyone can join at no cost. The announcements are always signed with the Microsoft Security Response Center PGP key, which is available from https://www.microsoft.com/technet/security/MSRC.asc, with a certificate from the Verisign/RSA Secure Server certification authority used in the https authentication process. The patch files themselves have digital signatures using Authenticode. Typically, the signature is made via a "Digital ID Class 3 - Microsoft Software Validation v2" certificate issued to Microsoft by the VeriSign Commercial Software Publishers certification authority. If a patch file is altered, viewing that patch file's "Digital Signature Details" will show an error stating, for example, "This digital signature is invalid.", "This digital signature is not valid." or "The integrity of the certificate which signed this file cannot be guaranteed. The certificate may be corrupted or has been tampered with.", depending on which Windows operating system is being used and on the nature of the change to the patch file. However, a modified patch file can in some cases still be executed and have its usual effects on the system. For example, on Windows 98 Second Edition, if a copy of
http://download.microsoft.com/download/win98/update/8070/w98/EN-US/259728USA8.EXE
is altered to display the text "This program installs the Windows 98 Q259782 Update." rather than "This program installs the Windows 98 Q259728 Update.", the program will still execute normally. This suggests that it is important for the user to examine the digital signature, since otherwise an attacker could trick the user into installing an unintended patch.

Descriptions of patches are available at URLs under https://www.microsoft.com/ (again, with a Verisign/RSA certificate used in the authentication). The patches themselves are obtained via URLs starting with http://download.microsoft.com/. The download.microsoft.com site does not support https; however, the availability of https would not add substantial value to patch authentication because of the individual signatures on the files.

NetBSD

Security patches for NetBSD are announced via mailing lists including netbsd-announce@netbsd.org. The announcements are PGP signed using the security-officer@netbsd.org PGP key -- the signatures on this key appear to be sufficient for it to be well trusted in terms of the PGP trust model. The announcements in some cases include full patch source code. In other cases, there is a patch file distributed via ftp; however, the patch file is always accompanied by a detached PGP signature, made using the security-officer@netbsd.org PGP key.

Patches consist of source code changes, and the base source code (ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-1.5/source/sets/) is apparently not itself PGP signed. The source availability provides an opportunity for source-code review by the user. Also, persons concerned about authentication of the online distribution have the option of purchasing the base distribution (including its source code) on CDROM. This does not address all distribution security concerns but does eliminate some possible mechanisms for trojan-horse distributions that are specific to the network distribution method.

Source code can also be obtained via anonymous CVS over ssh from a few different servers. This would constitute an authenticated mechanism for obtaining the source code, if the ssh public keys for these servers were obtainable via some authenticated means. This issue has been (indirectly) mentioned on the netbsd-help mailing list (e.g., see http://mail-index.netbsd.org/netbsd-help/2000/01/12/0015.html) but instructions for obtaining these ssh public keys do not seem to be available at http://www.netbsd.org/Sites/net.html#anoncvs

Novell / Netware

Patches for Novell products can be located starting from https://support.novell.com/filefinder/, with a certificate from the Equifax Secure certification authority providing authentication.

The patch contents are available from other servers (e.g., fdl.novell.com and ftp.novell.co.jp) only via http and ftp; the patch contents are apparently not available via https from any server.

The patches are distributed as executable programs that serve as an ARJ archive self-extractor. When the program corresponding to a valid patch is run, the program output includes 

  *** Valid ARJ-SECURITY envelope signature:
  *** Novell, Inc. R#1000026
If the patch file has been modified (at least for some types of modification), the output instead includes 
  Damaged self-extractor. Could be virus damage.
and no files are extracted. This method of validating the patch files does not provide useful authentication against some realistic attacks. For example, an attacker could write his own program (one that does not implement any ARJ functionality) that would create trojan-horse files with the correct names and sizes, and provide exactly the same output that would be provided by a real ARJ self-extractor.

Authentication would be improved if the user examined the patch file's ARJ digital signature by using a separate ARJ program obtained from a trusted source. The Novell web site does not appear to advise users to run a separate program for this purpose.

The patch files do not have other types of accompanying digital signatures (e.g., detached PGP signatures).

The banner provided by the ftp.novell.co.jp server is 

  220 support FTP server (Version wu-2.4.2-academ[BETA-12](2) Wed Mar 4 06:17:11 JST 1998) ready.
This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of
https://www.cert.org/advisories/CA-2000-13.html
which (without knowledge of other details of the ftp server) would suggest some likelihood that it is vulnerable to remote compromise. We sent the server some test "site exec" strings to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet ftp.novell.co.jp 21
  220 support FTP server (Version wu-2.4.2-academ[BETA-12](2) Wed Mar 4 06:17:11 JST 1998) ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %1$08x
  200-00000000
  200  (end of '%1$08x')
  SITE EXEC %2$08x
  200-0004c220
  200  (end of '%2$08x')
  SITE EXEC %3$08x
  200-81010100
  200  (end of '%3$08x')
  SITE EXEC %4$08x
  200-0000ff00
  200  (end of '%4$08x')
  SITE EXEC %5$08x
  200-ef589018
  200  (end of '%5$08x')
where 00000000, 0004c220, 81010100, 0000ff00, and ef589018 were apparently successive values in the ftpd process memory.

OpenBSD

Security patches for OpenBSD are announced via the web page http://www.openbsd.org/security.html. There is a mailing list named security-announce@openbsd.org; however, patches announced on that web page are not always also announced via that mailing list. Thus, persons running an OpenBSD release should be checking that web page in order to find out about the latest patches.

The patch files themselves are available via ftp in directories such as ftp://ftp.openbsd.org/pub/OpenBSD/patches/2.8/common/ and consist of source-code diff output along with build instructions. The patch files on that server are not accompanied by PGP signatures or other forms of cryptographic digital signatures.

Multiple versions of the source code for any file (including the source code incorporating the latest security patches) can, however, be obtained via anonymous CVS over an ssh connection from several servers including anoncvs6.usa.openbsd.org. The www.openbsd.org site has a link to an archive of the openbsd-announce mailing list. One of the messages in that archive contains an announcement of the ssh public key for anoncvs6.usa.openbsd.org -- the URL for that announcement is http://www.sigmasoft.com/~openbsd/archive/openbsd-announce/200007/msg00002.html. The announcement is PGP signed, using a key that appears to be well trusted in terms of the PGP trust model. Thus, anonymous CVS can be used an as authenticated mechanism for verifying source code obtained from anoncvs6.usa.openbsd.org.

This does not, however, imply that the service on anoncvs6.usa.openbsd.org constitutes an authenticated mechanism for obtaining the OpenBSD security patches, for two reasons. First, the OpenBSD project has not made any statement that an authenticated patch-distribution service exists. For example, there is no announcement stating that the entirety of all procedures used to transport source code from its original authors to anoncvs6.usa.openbsd.org is authenticated. Second, the OpenBSD project has not issued documentation about how to extract source-code differences from anonymous CVS servers in a way that will exactly match an OpenBSD security patch. Thus, the capability for verifying a security patch against the anonymous-CVS source distribution may be of limited usefulness to non-expert users.

In general, the OpenBSD project has not found enough value in PGP signatures to use them in conjunction with their distribution of security patches. There has been discussion within the OpenBSD community about using X.509 certificates to provide digital signatures of OpenBSD packages. The OpenBSD project's efforts in the security-patch area focus on trying to correct security problems comprehensively throughout their source tree, and focus on trying to ensure the host integrity of the servers used for distributing the patches and other source code. The OpenBSD project does not view source-code authentication as useful enough to divert resources away from these efforts.

The binary distribution files for an OpenBSD software release (e.g., from ftp://ftp.openbsd.org/pub/OpenBSD/2.8/) do not have PGP signatures or other types of cryptographic authentication. The OpenBSD project encourages users to obtain the release distribution (which includes the complete source code) by purchasing a CDROM. This does not address all distribution security concerns but does eliminate some possible mechanisms for trojan-horse distributions that are specific to the network distribution method.

Anonymous CVS can also be used to obtain the complete OpenBSD source code. Again, if anoncvs6.cvs.openbsd.org is the server chosen, the user obtains whatever degree of benefit is provided by that server's authenticated ssh public-key distribution. Even if the source code were obtained via an unauthenticated mechanism, there is an opportunity for source-code review by the user.

Red Hat

Security patches for Red Hat Linux are announced via mailing lists including redhat-watch-list@redhat.com. In the past, there have been PGP-signed announcements sent to this list; however, at present the messages are not being PGP signed. The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files themselves have embedded signatures using PGP. The security@redhat.com signing key is used; this key is available from https://www.redhat.com/about/redhat2.asc, with https authentication provided by a certificate issued by the Verisign/RSA Secure Server certification authority.

The security advisories are also available via this https service, at URLs beginning with https://www.redhat.com/support/errata/. The advisories link to patch files that are available via ftp (the patch files are apparently not available via https).

Red Hat also supports a patch service called the Red Hat Network. It is unclear whether future use of this service will be available only on a for-fee basis. https://www.redhat.com/products/network/ states that "Red Hat Linux 7 users are now eligible for a free trial of Red Hat Network services for up to 5 systems for a limited time" whereas https://www.redhat.com/network/service/faq_whatisrhn.html states that "The basic service will always be free to Red Hat members". A description of the Red Hat Network at https://www.redhat.com/network/service/faq_priv.html indicates that "All transactions are encrypted using a Secure Sockets Layer (SSL) connection". Thus, this may provide another authenticated mechanism for obtaining patches.

Most Red Hat packages are open source and this provides an opportunity for source-code review by the user.

The security advisories include a statement that 

  You can verify each package with the following command:
      rpm --checksig  <filename>
however, this type of verification does not provide authentication since it does not display any information about who signed the package. A user may have a public keyring containing a key of someone whom that user would not trust for signatures of RPM files. In that case, authentication can be provided by instead using the syntax "rpm --checksig -v <filename>".

SCO / OpenServer, UnixWare

Security patches for SCO products are announced via the web site http://www.sco.com/security/. Brief patch descriptions can be obtained via https, with authentication provided by a server certificate issued by the Verisign/RSA Secure Server certification authority, if the URL https://www.sco.com/security/ is used instead. The full patch descriptions and patch contents are available only via ftp, on the server ftp.sco.com.

The patch files are not accompanied by PGP signatures or other types of cryptographic digital signatures. A page on SCO's web site does, however, suggest that there is value in using PGP signatures. It says "Obtain the TCP Wrapper package for OpenServer 5 systems from: http://www.sco.com/security/sectools_utils.html. Note: Check the PGP signature file of the package as TCP Wrappers have been subject to Trojan horse attacks in the past."

SCO has no current plans to provide digital signatures of its patch files, or to provide access to patch files via https. On 2 August 2000, however, the SCO Server Software Division was acquired by Caldera. This acquisition may, at some point, lead to changes in the availability of SCO patch authentication. (Caldera currently, for example, includes PGP signatures in its OpenLinux packages.)

The banner provided by the ftp.sco.com server is 

  220 diana FTP server (Version 2.1WU(1)) ready.
This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of
https://www.cert.org/advisories/CA-2000-13.html
which (without knowledge of other details of the ftp server) would suggest some likelihood that it is vulnerable to remote compromise. We sent the server some test "site exec" strings to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet ftp.sco.com 21
  220 diana FTP server (Version 2.1WU(1)) ready.
  USER ftp
  331 Guest login ok, send e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %1$08x
  200-0000000d
  200  (end of '%1$08x')
  SITE EXEC %2$08x
  200-00000002
  200  (end of '%2$08x')
  SITE EXEC %3$08x
  200-000282c8
  200  (end of '%3$08x')
  SITE EXEC %4$08x
  200-76726573
  200  (end of '%4$08x')
  SITE EXEC %5$08x
  200-6374652f
  200  (end of '%5$08x')
  ...
  SITE EXEC %55839$08x
  200-0000196c
  200  (end of '%55839$08x')
where 0000000d, 00000002, 000282c8, 76726573, and 6374652f were apparently successive values in the ftpd process memory.

SGI / IRIX

Security patches for IRIX are announced via mailing lists including one named wiretap, which anyone may join at no cost. The announcements are always PGP signed, using the PGP key of the SGI Customer Security Coordinator <agent99@csd.sgi.com>. The signatures on this key appear to be sufficient for it to be well trusted in terms of the PGP trust model. Also, SGI's web site has a copy of the PGP key and includes a telephone number that can be used by customers wishing to verify this PGP key. The signed announcements include MD5 checksums of the patch files, thus providing authentication of the patch contents. Copies of the signed announcements are also available at http://www.sgi.com/support/security/advisories.html.

Slackware Linux

Security patches for Slackware Linux are announced via the slackware-security@slackware.com mailing list. The patches themselves are available via ftp from ftp.slackware.com and other sites. The "patch" typically consists of a new .tgz file that has the security fix incorporated into it.

Neither the patch files nor the announcements are accompanied by PGP signatures or other types of cryptographic digital signatures. Also, the patches are apparently not available on an https site.

The original set of files compromising a Slackware release is also not accompanied by any digital signatures. However, persons concerned about authentication of the online distribution have the option of purchasing the release itself (including its source code) on CDROM. This does not address all distribution security concerns but does eliminate some possible mechanisms for trojan-horse distributions that are specific to the network distribution method.

Most Slackware packages are open source and this provides an opportunity for source-code review by the user.

Sony / NEWS-OS

Security patches for NEWS-OS are distributed via the ftp server ftp.sony.co.jp. The packages are in the AT&T System V Release 4 pkg Datastream format, and do not include PGP signatures or other types of digital signatures. Also, the patches are apparently not available via https, and thus customers cannot use the authentication provided by a server certificate to help ensure that they are obtaining an unaltered patch issued by Sony.

The help text provided by the ftp.sony.co.jp server is 

  214-The following commands are recognized (* =>'s unimplemented).
     USER    PORT    STOR    MSAM*   RNTO    NLST    MKD     CDUP 
     PASS    PASV    APPE    MRSQ*   ABOR    SITE    XMKD    XCUP 
     ACCT*   TYPE    MLFL*   MRCP*   DELE    SYST    RMD     STOU 
     SMNT*   STRU    MAIL*   ALLO    CWD     STAT    XRMD    SIZE 
     REIN*   MODE    MSND*   REST    XCWD    HELP    PWD     MDTM 
     QUIT    RETR    MSOM*   RNFR    LIST    NOOP    XPWD 
  214 Direct comments to ftp-bugs@smcsc4.
suggesting that it may have an implementation based in part on wu-ftpd.

This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of

https://www.cert.org/advisories/CA-2000-13.html
which (without knowledge of other details of the ftp server) would suggest some likelihood that it is vulnerable to remote compromise. We sent the server some test "site exec" strings to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet ftp.sony.co.jp 21
  220 ftp1.sony.co.jp FTP server ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %1$08x
  200-7ffff550
  200  (end of '%1$08x')
  SITE EXEC %2$08x
  200-00000000
  200  (end of '%2$08x')
  SITE EXEC %3$08x
  200-0061696e
  200  (end of '%3$08x')
  SITE EXEC %4$08x
  200-0000002f
  200  (end of '%4$08x')
  SITE EXEC %5$08x
  200-00000002
  200  (end of '%5$08x')
where 7ffff550, 00000000, 0061696e, 0000002f, and 00000002 were apparently successive values in the ftpd process memory.

Sun Microsystems / Solaris

Sun maintains a Customer Warning System mailing list that is used for announcements of new Solaris security patches. Anyone may join this mailing list and there is no cost. Messages to this mailing list are always PGP signed -- the signatures on the signing key used appear to be sufficient for it to be well trusted in terms of the PGP trust model. The signed announcements do not contain checksums of the patch files, so the announcement itself does not contribute to authentication of the patch data. The patches themselves are available from ftp://sunsolve.sun.com/pub/patches/ and are not accompanied by PGP signatures or other types of cryptographic digital signatures. The patches are also available via http://access1.sun.com/. In some cases, Sun has patches available for download via https, with a certificate from the VeriSign International Server certification authority used in the authentication process -- for example, this is true for patches at URLs beginning with
https://www.sun.com/software/graphics/OpenGL/patches/
Our limited investigation of this suggests that https patch download is available only for patches to the OpenGL graphics applications environment.

The banner provided by the sunsolve.sun.com ftp server is 

  220 sunsolve6 FTP server (Version wu-2.6.0(3) Wed Jan 5 15:02:27 MST 2000) ready.
This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of
https://www.cert.org/advisories/CA-2000-13.html
which (without knowledge of other details of the ftp server) would suggest some likelihood that it is vulnerable to remote compromise. We sent the server some test "site exec" strings to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet sunsolve.sun.com 21
  220 sunsolve6 FTP server (Version wu-2.6.0(3) Wed Jan 5 15:02:27 MST 2000) ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %1$08x
  200-00000000
  200  (end of '%1$08x')
  SITE EXEC %2$08x
  200-00068b16
  200  (end of '%2$08x')
  SITE EXEC %3$08x
  200-00067728
  200  (end of '%3$08x')
  SITE EXEC %4$08x
  200-0005b7fc
  200  (end of '%4$08x')
  SITE EXEC %5$08x
  200-000658bc
  200  (end of '%5$08x')
  % telnet sunsolve.sun.com 21
  220 sunsolve6 FTP server (Version wu-2.6.0(3) Wed Jan 5 15:02:27 MST 2000) ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 Guest login ok, access restrictions apply.
  SITE EXEC %66487$08x 
  200-03000173
  200  (end of '%66487$08x')
where 00000000, 00068b16, 00067728, 0005b7fc, and 000658bc were apparently successive values in the ftpd process memory. Compared to other servers, the ftp connection to sunsolve.sun.com would more frequently die if multiple "site exec" commands were sent within a single session. One conceivable, although it would seem unlikely, explanation is that the ftpd is able to detect some types of anomalous "site exec" behavior, and is terminating ftp connections on that basis.

This "site exec" behavior on the sunsolve.sun.com ftp server was first reported to security-alert@sun.com on 25 September 2000. Our reports about this have resulted in replies from Sun, but we have not yet heard back about any plans for addressing the issue. We realize that Sun may be using a version of wu-ftpd that they modified to support multiple levels of login access, and that this may introduce some complications into updating of wu-ftpd.

BindView suggests that Sun customers who pay for support services (e.g., SunSpectrum Platinum Support) and that have concerns about patch-integrity issues contact Sun to inquire about the status of this ftp server. It would likely be more productive to initiate the inquiry through the mechanism that Sun provides in conjunction with the specific support offering (e.g., a telephone number that reaches the support team assigned to one's account).

SuSE

Security patches for SuSE Linux are announced via the mailing list named suse-security-announce and via the web page http://www.suse.com/us/support/security/. The announcements are PGP signed using the "SuSE Security Team <security@suse.de>" PGP key -- the signatures on this key appear to be sufficient for it to be well trusted in terms of the PGP trust model. Also, for future SuSE Linux distributions, SuSE plans to include its PGP public keys on the CDROM.

The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files themselves do not have embedded signatures using PGP. However, the PGP signed announcements contain MD5 checksums and these can be used for authentication of the contents of the .rpm files. SuSE plans to change its RPM packaging process soon such that the packages are signed.

Most SuSE packages are open source and this provides an opportunity for source-code review by the user.

Trustix

Security patches for Trustix are announced via the tsl-announce@trustix.com mailing list. The announcements contain MD5 checksums of the packages, but the announcements do not have PGP signatures or other types of digital signatures. The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files have embedded PGP signatures. The signing key used is "Trustix (TSL sign key) <tsl@trustix.com>". This key is self-signed and does not have any other signatures, and therefore this key is apparently not one that would be considered well trusted in terms of the PGP trust model. (The copy of the key that we found was included in the package gnupg-1.0.1-5tr.i586.rpm. The key was not available via any of the PGP key servers that we tried.)

Also, apparently neither this key nor the Trustix packages are available via https. Trustix AS does operate at least one https site, https://eshop.trustix.no/, which provides authentication via a certificate from the Thawte Server certification authority. However, that site apparently cannot be used to download Trustix packages.

These observations do not imply that the key is inappropriate for use in security announcements.

The Trustix packages are open source and this provides an opportunity for source-code review by the user.

TurboLinux

TurboLinux security patches are announced via the TL-Security-Announce@www.turbolinux.com mailing list. The messages are PGP signed using the key secmon@mail.turbolinux.com. This key is available from https://www.turbolinux.com/security/tlgpgkey.asc, with https authentication provided by a certificate issued by the Thawte Server certification authority.

The "patch" actually consists of a new package file known as a ".rpm" file. The .rpm package files themselves sometimes have embedded signatures using PGP, again with the secmon@mail.turbolinux.com signing key.

However, in some cases the packages do not actually have a PGP signature. For example, the TL-Security-Announce xchat TLSA2000022-1 announcement indicated that the new RPM was available at ftp://ftp.turbolinux.com/pub/updates/6.0/xchat-1.4.3-1.i386.rpm, and that "These packages are GPG signed by Turbolinux for security." The RPM is apparently instead located at ftp://ftp.turbolinux.com/pub/updates/6.0/RPMS/ and does not have a signature.

The announcement does, however, include the MD5 checksum of the .rpm file, and since the announcement is PGP signed, this can be used for authentication of the RPM.

Most TurboLinux packages are open source and this provides an opportunity for source-code review by the user.

WindRiver / VxWorks

WindRiver's description of the update process for its VxWorks operating system is provided on the web page
http://www.windriver.com/products/html/fieldupgrader_ds.html
The update process makes use of a product named GoAhead FieldUpgrader. The description mentions that "GoAhead FieldUpgrader uses the Digital Signature Standard (DSS) to secure communication between the server and remote devices." The update mechanism involves a target device polling a GoAhead UpgradeServer for the existence of an update. When the server indicates that the upgrade is available, the target device downloads and applies the upgrade. It was not clear whether the device has a copy of the GoAhead UpgradeServer's DSS public key, obtained via some trusted source separate from the GoAhead UpgradeServer, so that the device is able to authenticate the upgrade.

Some downloadable files associated with WindRiver's products are available via ftp://ftp.windriver.com/. The help text provided by the ftp.windriver.com server is 

  214-The following commands are recognized (* =>'s unimplemented).
     USER    PORT    STOR    MSAM*   RNTO    NLST    MKD     CDUP 
     PASS    PASV    APPE    MRSQ*   ABOR    SITE    XMKD    XCUP 
     ACCT*   TYPE    MLFL*   MRCP*   DELE    SYST    RMD     STOU 
     SMNT*   STRU    MAIL*   ALLO    CWD     STAT    XRMD    SIZE 
     REIN*   MODE    MSND*   REST    XCWD    HELP    PWD     MDTM 
     QUIT    RETR    MSOM*   RNFR    LIST    NOOP    XPWD 
  214 Direct comments to postmaster@wrs.com.
suggesting that it may have an implementation based in part on wu-ftpd.

This server does expansion of format strings that are passed via the "site exec" command as described at the beginning of section I of

https://www.cert.org/advisories/CA-2000-13.html
which (without knowledge of other details of the ftp server) would suggest some likelihood that it is vulnerable to remote compromise. We sent the server some test "site exec" strings to confirm this format-string property, and of course we did not send it any input data that would constitute a login attempt or other exploit: 
  % telnet ftp.windriver.com 21
  220 scylla FTP server ready.
  USER ftp
  331 Guest login ok, send your complete e-mail address as password.
  PASS guest@
  230 User ftp logged in.  Access restrictions apply.
  SITE EXEC %1$08x
  200-00000000
  200  (end of '%1$08x')
  SITE EXEC %2$08x
  200-000700e0
  200  (end of '%2$08x')
  SITE EXEC %3$08x
  200-81010100
  200  (end of '%3$08x')
  SITE EXEC %4$08x
  200-00000000
  200  (end of '%4$08x')
  SITE EXEC %5$08x
  200-00081558
  200  (end of '%5$08x')
  ...
  SITE EXEC %100000$08x
  200-6f6e2064
  200  (end of '%100000$08x')
where 00000000, 000700e0, 81010100, 00000000, and 00081558 were apparently successive values in the ftpd process memory.

Credits

Responses were received from persons associated with the following vendors: Caldera, Compaq, Conectiva, Debian, FreeBSD, Hewlett-Packard, IBM, WireX, Microsoft, OpenBSD, SCO, SGI, Sun Microsystems, and SuSE. Thank you all very much for responding.

The following pieces of related previous work may be of interest to readers:
Security Issues of Auto-upgrades (from freshmeat.net)
Xato commentary on MS security bulletins
SSL Server Security Survey, by Eric Murray
Use of Akamai hosts to circumvent SSL server authentication, by Kevin Fu
Why Digital Signatures Are Not Signatures, by Bruce Schneier
Introduction to Public-Key Cryptography, from Netscape
dsniff, by Dug Song
The End of SSL and SSH?, by Kurt Seifried