Comparison of Rating Systems

Figure I.1: Ratings Could Not Be Easily Compared

GAO:  Ratings Could Not Be Easily Compared

• Rating scales and descriptions varied
  
  
• Weiss placed less reliance on analysts' judgment than other raters
  
  
• Coverage differed -- only Weiss rated most insurers
  
  
• Distributions varied -- Weiss and Moody's assigned fewer ratings at the top of their scales than other raters

Ratings Could Not Be Easily Compared

The five rating agencies that rate life/health insurers -- Best, D&P, Moody's, S&P and Weiss -- did not all use the same approach and methods to rate insurer financial health during the period of our analysis.

The rating agencies used different symbols and numbers of ratings in their rating scales. In some cases, the same letter rating was used by various raters. However, it did not necessarily occupy the same relative position in the respective rating scales. For example, an "A+" was Weiss' highest rating; Best's second-highest score; and D&P's and S&P's fifth-highest score.

The same letter rating also had different descriptions from the various raters. The "A+" rating was described as "excellent" by Weiss, "superior" by Best and "good" by S&P.¹ Even the same letter rating within one rater's system may have had a different meaning at two points in time. During the time that we did our work, Best changed its rating system twice, both adding new ratings and changing the definition of existing ratings. In 1992, it added three new ratings, dropping its "A+" rating from first to second place. And, in 1994, it adopted the two-category "secure/vulnerable" classification used by most of the other raters.

All rating agencies, with the exception of Weiss, said that they placed a great deal of reliance on analysts' judgment both in assessing an insurer's management and in adjusting an insurer's rating at the end of the rating process. In contrast, while Weiss used judgment to create and modify the mathematical model that it used to determine the ratings, Weiss did not use judgment to assess management or to alter the results produced by the model for an insurer at the end of the rating process.

The agencies also differed in the number of life/health insurers they rated. Weiss, which does not charge insurers a fee, rated most insurers. The other rating agencies, which usually charge fees, rated fewer insurers. However, because all agencies generally rated the largest insurers, they each covered at least 50 percent of the life/health insurance industry's assets.

The five raters also varied in their assignment of ratings to insurers. We compared the agencies' distribution of ratings both for (1) insurers rated by each agency individually and (2) insurers rated in common between agencies, two agencies at a time. In both of these comparisons, Weiss and Moody's were less likely than the other raters to assign insurers their top ratings.

¹ Although D&P's full description for "A+" was similar to that of S&P, D&P did not use a single word descriptor similar to S&P's "good".

Figure I.2: Rating Scales Differed

GAO:  Rating Scales Differed

Ratings Bands Weiss a Best a,b S&P c Moodys D&P Secure 1 A+,A,A- A++,A+ AAA Aaa AAA   2 B+,B,B- A,A- AA+,AA,AA- Aa1,Aa2,Aa3 AA+,AA,AA-   3 C+,C,C- B++,B+ A+,A,A- A1,A2,A3 A+,A,A-       B,B- BBB+,BBB,BBB- Baa1,Baa2,Baa3 BBB+,BBB,BBB Vulnerable 4 D+,D,D- C++,C+ BB+,BB,BB- Ba1,Ba2,Ba3 BB+,BB,BB-       C,C- B+,B,B- B1,B2,B3 B+,B,B-   5 E+,E,E- D CCC,(CC,C) Caa,Ca,C CCC+,CCC,CCC-     F E,F (D),R   DD

^a Weiss and Best use additional symbols to designate that they recognize an insurer's existence but do not provide a rating. These symbols are not included in this table.

^b Best added the A++, B++ and C++ ratings in 1992. In 1994, Best classified its ratings into "secure" and "vulnerable" categories, changed the definition of its "B" and "B-" ratings from "good" to "adequate" and assigned these ratings to the "vulnerable" category. This table contains GAO's assignment of Best's ratings to bands based on our interpretation of their rating descriptions prior to 1994.

^c S&P discontinued CCC "+" and "-" signs, CC, C and D ratings and added the R rating in 1992.

Source: GAO.

Rating Scales Differed

The rating agencies used different symbols and different numbers of ratings in their scales. The number of ratings used ranged from 15 (Best) to 20 (D&P). Weiss' top rating was "A+", Best's "A++", S&P's and D&P's "AAA" and Moody's "Aaa."

Although the same letter ratings were used by several raters, they did not necessarily have the same meaning. For example, Weiss' "A+" "excellent" rating was its top rating. Best called its "A+" rating "superior," but it was its second-highest score. And S&P's "good" "A+", D&P's "A+" and Moody's "good" "A1" ratings were their fifth-highest scores.

Three of the rating agencies--S&P, Moody's and D&P--divided their rating scales into two categories labeled "secure" and "vulnerable."² On the basis of their descriptions, we assigned Weiss' and Best's ratings to these two categories. Weiss agreed with our assignment. Best told us that they were considering revising their rating scale to conform to the two-category "secure/vulnerable" designation used by most of the other raters and that their empirical analysis had revealed that their "B" and "B-" ratings were similar to the other agencies' "vulnerable" ratings. However, Best did not revise their rating scale for the period they supplied data for our analysis. We assigned their "B" and "B-" "good" ratings to the "secure" category because we believe that a consumer would not understand a "good" insurer to be "vulnerable."

Rather than attempting to force the various rating designations to correspond letter rating by letter rating, we assigned them into five bands based on the similarity of their descriptions. As shown in figure I.2, we divided "secure" ratings into three bands, labeled "1" through "3" and "vulnerable" ratings into two bands, labeled "4" and "5." Thus, adjacent horizontal cells in figure I.2 contain ratings whose descriptions we determined were similar. However, as we previously noted, the descriptions were not precisely equivalent. Thus, these finer distinctions reflect our view of how the different ratings corresponded.³

² While Moody's and D&P may use different labels such as "investment grade/noninvestment grade" and "strong/weak", they told us that these labels have the same meaning as the "secure/vulnerable" labels used by S&P.

³ S&P, Moody's and D&P told us they intend for their ratings, which occupy similar positions in their respective scales, to be equivalent.

Figure I.3: Only Weiss Rated More Than Half of All Life/Health Insurers

GAO:  Only Weiss Rated More Than Half of All Life/Health Insurers

Note 1: Data are for June 30, 1992.

Note 2: Percent rated "vulnerable": S&P 0.2 percent (0.4 percent asset-weighted); Moody's 0.2 percent (1.5 percent asset-weighted); D&P 0.0 percent (0.0 percent asset-weighted).

Note 3: Our computation of S&P's coverage included only their Claims-Paying Ability (CPA) ratings and excluded their Qualified Solvency Ratings (QSR). Had S&P's QSR ratings been included, their coverage would have increased to 46.4 percent (96.3 percent asset-weighted).

Source: GAO.

Only Weiss Rated More Than Half of All Life/Health Insurers

Figure 1.3 shows the percentage of life/health insurers rated by each of the raters, both in number and weighted by asset size. As of June 30, 1992, we identified 1,963 life/health insurers. Weiss rated 1,449--over 70 percent of the universe we identified, as compared to 795 rated by Best--about 40 percent.⁴ The other three raters covered 12 percent or less each. However, because S&P, Moody's and D&P generally rated the larger insurers, they each covered at least 50 percent of the life/health insurance industry's assets.

Figure 1.3 also shows the division of ratings into "secure" and "vulnerable" categories. Although both S&P and Moody's assigned "vulnerable" ratings, the amounts were generally too small to register on the charts above.

⁴ Weiss and Best provided some information about but did not rate an additional 16 percent (0.5 percent asset-weighted) and 29 percent (3.9 percent asset weighted) of life/health insurers, respectively.

Figure 1.4: Weiss and Moody's Assigned Fewer Top Ratings

GAO:  Weiss and Moody's Assigned Fewer Top Ratings

Note 1: Data are for June 30, 1992.

Note 2: Each agency's results are based on the distribution of ratings for all insurers rated by that agency.

Source: GAO.

Weiss and Moody's Assigned Fewer Top Ratings

We compared agencies' distributions of ratings both for (1) insurers rated by each agency individually and (2) insurers rated in common between agencies, two agencies at a time. Because the agencies did not all rate the same insurers, their rating distributions in the first comparison are likely to be different. Indeed, the distributions from the five agencies differed considerably. Best's, S&P's and D&P's distributions contained proportionately more high ratings than those of Weiss and Moody's. Weiss' ratings were consistent with a normal or bell-shaped distribution, with most of the ratings falling in the middle of the scale. However, Best, which rated fewer insurers than Weiss, but more insurers than S&P, Moody's and D&P, had the highest percentage of ratings in the first band. Although higher than Weiss, Moody's percentage of its rating distribution in the first band was lower than that of Best, D&P and S&P.

Figure I.4 shows rating data from June 30, 1992, for insurers rated by each agency individually. When the median ratings for the various raters are placed within the roughly equivalent bands of figure I.2 on page 8, Weiss' median "C-" rating falls in the third band, at the bottom of the "secure" range, while the other raters' median ratings fall in the second band, in the middle of the "secure" range. Similarly, figure I.4 also shows that Weiss assigned the lowest percentage of ratings to the higher rating categories, followed by Moody's. Other raters assigned ratings in the higher bands two to nine times more often than Weiss.

As explained above, some of the variability in the raters' distributions may be because they did not rate the same insurers. To remove this possible source of difference, we also compared the assignment of ratings between agencies only for insurers they rated in common, as shown in the next few pages.

Figure I.5: Raters Usually Agreed Whether an Insurer Was "Secure" or "Vulnerable"

GAO:  Raters Usually Agreed Whether an Insurer Was "Secure" or "Vulnerable"

• Raters agreed from 89% to 100% of the time when placing insurers in "secure" or "vulnerable" categories
  
  
• When raters disagreed, Weiss assigned "vulnerable" ratings more often

We compared the five raters' assignment of ratings to insurers which they rated in common, comparing two raters at a time. At the broadest level, we found that the raters usually agreed on whether the insurers were "secure" or "vulnerable." Using data from June 30, 1992, we found that they agreed from 89 percent to 100 percent of the time, depending on which two raters we compared.

When the raters disagreed about whether an insurer was "vulnerable," we found that Weiss was more likely to assign a "vulnerable" rating than the other raters.

Figure I.6: Assignment of Top Ratings: Weiss vs the Other Raters

GAO:  Assignment of Top Ratings: Weiss vs the Other Raters

Note 1: Data are for June 30, 1992.

Note 2: "Secure" ratings are contained in the first three bands. Shown here are ratings in the first and second bands. (See figure I.2 for a definition of the rating bands.)

Note 3: Information in this chart is based on pairwise comparisons of ratings for insurers rated in common by Weiss and the other rater.

Note 4: The numbers of insurers rated by both Weiss and another rater on June 30, 1992, are: 781 (Best), 217 (S&P), 78 (Moody's and 95 (D&P). Pairs of bars depict the percentages of these totals Weiss and the other rater assigned top ratings in the first two bands.

Source: GAO.

Assignment of Top Ratings: Weiss Versus the Other Raters

We then used the rating bands from figure I.2, page 8, to further compare the raters' assignment of ratings to insurers that they rated in common. We did this comparison two agencies at a time. Because Weiss rated the largest number of insurers and also differed the most from other raters in assignment of ratings, we show the cases where Weiss was compared to the other raters in figure I.6. When the bars in a pair are of similar height, this indicates that Weiss and the other rater assigned about the same percentage (and number) of top ratings to insurers they both rated. As figure I.6 shows, Weiss and Moody's assigned about the same percentage of ratings to either the first band or the first and second bands combined for insurers they rated in common. When we reviewed all the pairwise comparisons between the raters (not shown here), we found that Weiss and Moody's assigned fewer ratings in either the first band or the first and second bands than Best, D&P and S&P.