Wednesday, June 29, 2011

More proxy temperature reconstruction plots

This is a sequel to the previous post, where the list of data information, taken from the NOAA file, is printed. I have extended the time scale to 2000 years. This time I have used 41-year triangle smoothing. And I have experimented with inclusion and omission of instrumental temperatures.

This the plot over 2 millennia with HADCRUT3NH (with 11 year smoothing) in a background light grey.


More below the jump...



And here is the same plot, but without HADCRUT3, nor other reconstructions with instrumental components



And here is the 1 millenium plot, with smoothing and instrumental



and now the same without instrumental




Monday, June 27, 2011

Northern Hemisphere proxy plots - last millenium

This is the next in what may be a series of multiple time series plotted with animated gif's in the hope of greater visual clarity.

I found at NOAA a large table of proxy-based temperature reconstruction data. The background is described in this Wiki article. For the moment, I've plotted just the Northern Hemisphere reconstructions for the last millenium. For those who like that sort of thing, I've currently omitted the instrumental curve at the end. More plots may appear later here.
Update - I misspoke here. The Mann2008g and 2008h curves are a composite including instrumental temperatures. Crowley2003b also includes instrumental, but only up to 1993.
Unrelated - I have replaced the plot with one including a few more series from the same resource. I have use a consistent anomaly base period of 1936-1965 (the most recent 30-yr period common to all). The Oerlemans set is based on glaciers and is described as global rather than NH.

Here is the plot, unsmoothed. Below the jump is a summary of the data sets.


Update:  In the listing below, I have added a number beside each bolded name. This is the number in the NOAA table as extended with HADCRUT etc. It will be useful for anyone running the code and wanting to vary the subsets chosen.

Key data: 
93 HADCRUT3NH
TITLE: Land and Sea Temperature Anomalies Northern Hemisphere
CITATION: Brohan, P., J.J. Kennedy, I. Harris, S.F.B. Tett and P.D. Jones, 2006: Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850. J. Geophysical Research 111, D12106, doi:10.1029/2005JD006548
DESCRIPTION_SUMMARY: combined land and marine [sea surface temperature (SST) anomalies from HadSST2, see Rayner et al., 2006] temperature anomalies on a 5° by 5° grid-box basis. https://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3nh.txt
94 Loehle
TITLE: Northern Hemisphere Temperature Reconstructions
CITATION: Correction to: A 2000-Year Global Temperature Reconstruction Based on Non-Tree Ring Proxies C Loehle, JH McCulloch - Energy &# 38; …, 2008 - Multi-Science Publishing Co Ltd.
DESCRIPTION_SUMMARY: A 2000-YEAR GLOBAL TEMPERATURE RECONSTRUCTIONBASED ON NON-TREERING PROXIES https://www.ncasi.org/programs/areas/climate/LoehleE&E2007.csv
95 MBH98
TITLE: Global-scale temperature patterns and climate forcing over the past six centuries
CITATION: Global-scale temperature patterns and climate forcing over the past six centuries by: M. E. Mann, R. S. Bradley, M. K. Hughes Nature, Vol. 392 (1998), pp. 779-787. doi:10.1038/33859
DESCRIPTION_SUMMARY: Plotted here from archived MM05 data set
1 ammann2007
TITLE: Northern Hemisphere Average Annual Temperature Reconstruction
CITATION: Ammann, C.M. and E.R. Wahl. 2007. The importance of the geophysical context in statistical evaluations of climate reconstruction procedures. Climatic Change 85:71-88. DOI: 10.1007/s10584-007-9276-x. See also companion article by Wahl and Ammann: Climatic Change, 85:33-69 (2007) DOI: 10.1007/s10584-006-9105-7
DESCRIPTION_SUMMARY: Uses multiple proxy types, input into inverse regression-truncated EOF climate field reconstruction spanning entire globe at incomplete 5x5 deg grid. Only N. Hemisphere average is reported here.
13 briffa1998
TITLE: Northern Hemisphere Temperature Reconstructions
CITATION: Briffa, K.R., P.D. Jones, F.H. Schweingruber and T.J. Osborn. 1998. Influence of volcanic eruptions on Northern Hemisphere summer temperature over the past 600 years. Nature 393:450-455.
DESCRIPTION_SUMMARY: Derived from means of 383 maximum latewood density chronologies from the northern Boreal forest.
31 crowley2000a
TITLE: Northern Hemisphere Temperature Reconstruction
CITATION: Crowley, T.J. 2000. Causes of Climate Change Over the Past 1000 Years. Science 289:270-277.
DESCRIPTION_SUMMARY: Proxies used include tree-rings, pollen, oxygen isotopes, ice core, phenological records, historical records. Modification of reconstruction from Crowley, T.J., and T. S. Lowery. 2000. How Warm was the Medieval Warm Period. Ambio 29:51-54.
32 crowley2000b
TITLE: Northern Hemisphere Temperature Reconstruction: with instrumental records after 1860
CITATION: Crowley, T.J. 2000. Causes of Climate Change Over the Past 1000 Years. Science 289:270-277.
DESCRIPTION_SUMMARY: Proxies used include tree-rings, pollen, oxygen isotopes, ice core, phenological records, historical records. Modification of reconstruction from Crowley, T.J., and T. S. Lowery. 2000. How Warm was the Medieval Warm Period. Ambio 29:51-54, with the instrumental record from Jones, P., M. New, D. Parker, S. Martin, and I. Rigor. 1999. Surface Air Temperature and its Changes Over the Past 150 Years. Reviews of Geophysics 37:173-199 used after 1860.
33 darrigo2006a
TITLE: Northern Hemisphere Tree-Ring-Based Temperature Reconstruction: Standard
CITATION: D'Arrigo, R., R. Wilson, and G. Jacoby. 2006. On the long-term context for late twentieth century warming. Journal of Geophysical Research 111:D03103. DOI: 10.1029/2005JD006352.
DESCRIPTION_SUMMARY: Standard Reconstruction (negative-exponential or straightline curve fits). Tree-ring based reconstruction from 66 high elevation and latitudinal treeline North American and Eurasian sites.
34 darrigo2006b
TITLE: Northern Hemisphere Tree-Ring-Based Temperature Reconstruction: Regional Curve Standardization
CITATION: D'Arrigo, R., R. Wilson, and G. Jacoby. 2006. On the long-term context for late twentieth century warming. Journal of Geophysical Research 111:D03103. DOI: 10.1029/2005JD006352.
DESCRIPTION_SUMMARY: Regional Curve Standardization (RCS) Reconstruction. Tree-ring based reconstruction from 66 high elevation and latitudinal treeline North American and Eurasian sites.
40 huang2004
TITLE: Integrated Northern Hemisphere Surface Temperature Reconstruction
CITATION: Huang, S. 2004. Merging Information from Different Resources for New Insights into Climate Change in the Past and Future. Geophysical Research Letters 31:L13205. DOI: 10.1029/2004GL019781.
DESCRIPTION_SUMMARY: Reconstruction based on borehole temperatures, the 20th century meteorological record, and multi-proxy paleoclimatic records.
43 jones1998a
TITLE: Millennial Temperature Reconstructions: Northern Hemisphere
CITATION: Jones, P.D., K.R. Briffa, T.P. Barnett, and S.F.B. Tett. 1998. High-resolution Palaeoclimatic Records for the last Millennium: Interpretation, Integration and Comparison with General Circulation Model Control-run Temperatures. The Holocene 8:455-471.
DESCRIPTION_SUMMARY: tree rings, ice cores, corals, and historical documents
51 mann1999
TITLE: Northern Hemisphere Temperatures During the Past Millennium
CITATION: Mann, M.E., R.S. Bradley, and M.K. Hughes. 1999. Northern Hemisphere Temperatures During the Past Millennium: Inferences, Uncertainties, and Limitations. Geophysical Research Letters 26:759-762.
DESCRIPTION_SUMMARY: Proxies used include tree-rings, ice cores, corals, long historical records, and long instrumental data series. Extension over 1000 AD to 1399 AD of Mann, M.E., R.S. Bradley, and M.K. Hughes. 1998. Global-Scale Temperature Patterns and Climate Forcing Over the Past Six Centuries. Nature 392:779-787.
53 mann2003b
TITLE: 2,000 Year Hemispheric Multi-proxy Temperature Reconstructions: Northern Hemisphere
CITATION: Mann, M.E. and P.D. Jones. 2003. Global Surface Temperatures over the Past Two Millennia. Geophysical Research Letters 30:1820. DOI: 10.1029/2003GL017814.
DESCRIPTION_SUMMARY: Tree-rings, historical records, lake sediments, ice cores, fossil shells, and boreholes. Decadally-resolved series
57 mann2008a
TITLE: 2,000 Year Hemispheric and Global Surface Temperature Reconstructions: Northern Hemisphere: Land Only: Composite Plus Scale Method
CITATION: Mann, M.E., Z. Zhang, M.K. Hughes, R.S. Bradley, S.K. Miller, S. Rutherford, and F. Ni. 2008. Proxy-based reconstructions of hemispheric and global surface temperature variations over the past two millennia. Proceedings of the National Academy of Sciences 105:13252-13257. DOI:10.1073/pnas.0805721105.
DESCRIPTION_SUMMARY: Proxies include tree-ring, marine sediment, speleothem, lacustrine, ice core, coral, and historical documentary series. Composite reconstruction formed by averaging all validated reconstruction scenarios for the given reconstruction method and spatial target. Cf. page 13255 of original publication and Supporting Information Figures S5 and S6.
58 mann2008b
TITLE: 2,000 Year Hemispheric and Global Surface Temperature Reconstructions: Northern Hemisphere: Land and Ocean: Composite Plus Scale Method
CITATION: Mann, M.E., Z. Zhang, M.K. Hughes, R.S. Bradley, S.K. Miller, S. Rutherford, and F. Ni. 2008. Proxy-based reconstructions of hemispheric and global surface temperature variations over the past two millennia. Proceedings of the National Academy of Sciences 105:13252-13257. DOI:10.1073/pnas.0805721105.
DESCRIPTION_SUMMARY: Proxies include tree-ring, marine sediment, speleothem, lacustrine, ice core, coral, and historical documentary series. Composite reconstruction formed by averaging all validated reconstruction scenarios for the given reconstruction method and spatial target. Cf. page 13255 of original publication and Supporting Information Figures S5 and S6.
63 mann2008g
TITLE: 2,000 Year Hemispheric and Global Surface Temperature Reconstructions: Northern Hemishpere: Land Only: Error-In-Variables Method
CITATION: Mann, M.E., Z. Zhang, M.K. Hughes, R.S. Bradley, S.K. Miller, S. Rutherford, and F. Ni. 2008. Proxy-based reconstructions of hemispheric and global surface temperature variations over the past two millennia. Proceedings of the National Academy of Sciences 105:13252-13257. DOI:10.1073/pnas.0805721105.
DESCRIPTION_SUMMARY: Proxies include tree-ring, marine sediment, speleothem, lacustrine, ice core, coral, and historical documentary series. Error-In-Variables (EIV) based on RegEM algorithm. Composite reconstruction formed by averaging all validated reconstruction scenarios for the given reconstruction method and spatial target. Cf. page 13255 of original publication and Supporting Information Figures S5 and S6.
64 mann2008h
TITLE: 2,000 Year Hemispheric and Global Surface Temperature Reconstructions: Northern Hemisphere: Land and Ocean: Error-In-Variables Method
CITATION: Mann, M.E., Z. Zhang, M.K. Hughes, R.S. Bradley, S.K. Miller, S. Rutherford, and F. Ni. 2008. Proxy-based reconstructions of hemispheric and global surface temperature variations over the past two millennia. Proceedings of the National Academy of Sciences 105:13252-13257. DOI:10.1073/pnas.0805721105.
DESCRIPTION_SUMMARY: Proxies include tree-ring, marine sediment, speleothem, lacustrine, ice core, coral, and historical documentary series. Error-In-Variables (EIV) based on RegEM algorithm. Composite reconstruction formed by averaging all validated reconstruction scenarios for the given reconstruction method and spatial target. Cf. page 13255 of original publication and Supporting Information Figures S5 and S6.
68 moberg2005
TITLE: 2,000-Year Northern Hemisphere Temperature Reconstruction
CITATION: Moberg, A., D.M. Sonechkin, K. Holmgren, N.M. Datsenko and W. Karlén. 2005. Highly variable Northern Hemisphere temperatures reconstructed from low- and high-resolution proxy data. Nature 433:613-617.
DESCRIPTION_SUMMARY: Reconstruction calculated by combining low-resolution proxies with tree-ring data, using a wavelet transform technique.
70 oerlemans2005
TITLE: Global Glacier Length Temperature Reconstruction
CITATION: Oerlemans, J. 2005. Extracting a Climate Signal from 169 Glacier Records. Science 308, No:675-677.
DESCRIPTION_SUMMARY: Global temperature reconstruction based on glacier length records from 169 locations.
75 smith2006
TITLE: Northern Hemisphere Speleothem Temperature Reconstruction
CITATION: Smith, C.L., A. Baker, I.J. Fairchild, S. Frisia, and A. Borsato. 2006. Reconstructing hemispheric-scale climates from multiple stalagmite records. International Journal of Climatology 26(10):1417-1424.
DESCRIPTION_SUMMARY: annual, speleothems, northern hemisphere, stalagmite layer thickness from Scotland, Italy, and China.

Thursday, June 23, 2011

Steven Mosher's GHCN V3 R Package

It's now up at CRAN. You can find it here.

The Mac and Windows binaries are not there yet, but coming soon. But the reference manual, which is the only thing I have really looked at yet, is there.

I'll write more when I've been able to try it out (waiting for binaries). More information at Steven's blog.



Wednesday, June 22, 2011

Time series plots - using animation

In my posts so far in trying to find more readable ways of presenting multiple series plots, I've tried multi-colors, and varying spectral maps. With multi-colors the lines may be easier to distinguish but harder to follow.

Eli, in a comment in the first post, suggested making curves respond to the pointer rolling over. This needs Java programming, which I can't do. But it occurred to me that an animated gif would achieve something of the same effect.

If each curve becomes, for a period, a continuous black line, then its path can be traced easily. Between times, the dots will separate out the local features. I'll switch to this more topical example - JAXA Ice extent:




More visible time series plots

In my previous post I described the use of alternating colors to improve the readability of "spaghetti" plots of time series, especially for readers who had trouble distinguishing fine shades of color. I updated several times, so if you read it a while ago, you might like to check it again.

There was feedback, here and at Lucia's, from readers concerned about color-blindness, especially red-green, That got me thinking more about appropriate color schemes.

The benefit of three colors alternating, as I had, is that one can hope thatmost people could distinguish at least two of them, since they come from different parts of the rainbow(). But maybe that can be reinforced.

The downside to all this is that alternating colored lines are harder to follow by eye than single color.

Anyway, I've looked more into the R function rainbow(). It is just scanning the hue spectrum in the hsv() function. I'll talk more about HSV and RGB color numberings below. For the moment, this just makes possible a more flexible approach to the spectrum, which may help with color difficulties.

I've plotted here the same TSI example using different spectral ranges. You can click on any plot to see enlarged. The top blue mini-graph shows the part of the spectrum that is enhanced - more colors are chosen from the region where that function is higher. Below is a bar with the uniform spectrum, and below that, the spectrum actually chosen. The rest is as before. Below the jump come some thick line versions.






I'll be interested to hear if any of these color selections seem to be more easily discriminated.

Technical details - RGB, HSV etc

This gets into how colors are represented by numbers. The simplest model is RGB - three numbers representing the amount of red, green and blue. A notation common to many graphics platforms is the string "#rrggbb" where r,g,b are hex digits. So "#ff0000" is just red, "#bbbb00" is gold, and "#444444" is dark grey. Often two more digits are added to represent transparency - FF for none.

HSV is intended to be more in line with the way we perceive colors. Again it's a triple of numbers (in R on a scale of 0-1). H represents hue - like the familiar spectrum. S is saturation, and pretty much represents the amount of color at a given brightness. The brightness (as opposed to darkness) is given by the value (black=0, bright=1).

One thing to note in R is that HSV emulates the spectrum rather than generates it. The violet colors are generally created using reds with blue, so the high end doesn't help if you have trouble with red.

The following plot should make this clearer. Each bar shows the effect of varying one of h, s or v from 0 to 1.



R has various routines that support rgb etc. rgb(r,g,b) turns a triple (range 0-1) into a string for color. hsv(h,s,v) likewise creates a string. Then there are rgb2hsv etc. To make all these plots, I just used the hsv() function.

To get the modified spectrum, I decide on a function f(x) as shown in the top plots above. Then I invert and sum, and normalize to a 0:1 range. That gives a mapping vector i (length N) that moves rapidly through the unwanted colors. So hsv(i,1,1) gives the N colors to be used.

Friday, June 17, 2011

Cheerful colors for time series

Over at the Blackboard. Lucia was looking at how to get a good color scheme in R to show multiple time series. It's quite hard to get a set with good contrast.

I've been wondering about that too. It's a personal problem - my ability to distinguish color shades has decreased.

I've been dabbling with an alternative idea - stripy lines. Or at least alternating color segments. Then you don't have to rely on shades to make the distinction.

Lucia illustrated with some solar data from Leif Svalgaard. She used different dot-dash line styles to nelp make contrasts. I thought it would be really good to make these in alternating colors. You can do this by over-writing.

So here's what I came up with. Some may like it, some not. The lines are in principle more distinctive, but it's harder to see where they are going. Single contrasting colors are certainly better, if you can get enough of them.

Anyway, here's my plot. The R code is below the jump, and I'll put a zip file (TSIcolors.zip) with data on the doc repository. As Lucia noted, Leif's file just has blanks for missing data, so I edited the NA entries in.The colors are automatically and randomly chosen.

Update:
Peter O'Neill (oneillp) in comments  suggested using R-supplied palettes. I think this is better, specifically rainbow(). He also suggested a way to fix the line segments in legend, using seg.len. I found my legend() function would not take that as an argument. I also found that the problem with lines only applied when in jpeg or png mode. I couldn't find the bug, so I wrote my own legend routine - using a subset of the regular arguments. 
Update.  Replacing the above update. I've redone in the spirit of Peter's second comment. Instead of a new legend function, I use the values returned by the the standard oneto overwrite the line segs. I don't then need to use seg.len

Revised pictures and code below. 





Here's a plot with thicker lines. I couldn't get the legend lines right here:

And here is the (revised) code:

#  Program written by Nick Stokes to make multi-colored curves
# File from https://www.leif.org/research/TSI%20%28Reconstructions%29.txt
# Blanks have been converted to NA


w=t(read.table("leif.txt",skip=4,nrows=311)) 
N=dim(w)[1]-1  # Number of curves
x=w[1,]
cl=rainbow(N)
cl=matrix(cl[round(outer(1:N*3-3,0:2*N,"+")/3)%%N+1],N,3) ## Make orthog colors
# Now make dash patterns
   k1=1:N%%3+2;k2=1:N%%5+8; k=k1+k2;
   lt=matrix(paste(as.hexmode(c(241+k*0,k1*16+k2,15+k))),N,3)
   names=c("Hoyt","Leif","Dora","Wang","Lean","PMOD","ACRIM","TIM","DIARAD  ","Krivova")
   lw=4  # line width
### Now plotting
png("TSI1.png",width=800)
plot(range(x),range(w[1:N+1,],na.rm=T),type="n",xlab="Year",ylab="TSI")
#Now plot curves 3 times with 3 different colors and dash patterns

for(i in 1:N)for(j in 1:3)lines(x,w[i+1,], col=cl[i,j], lty=lt[i,j], lwd=lw)
# Now plots legend, also 3 times
L = legend(1970,1364.74, legend=names, cex=1.0, text.col=cl[,1], col="white",lty=lt[,1])
x=L$rect$left+c(0.02,0.3)*L$rect$w
for(i in 1:N) for(j in 1:3) lines(x,rep(L$text$y[i],2),col=cl[i,j],lty=lt[i,j],lwd=lw)

dev.off()

Wednesday, June 8, 2011

Effect of selection in the Wegman Report

The Wegman Report was a report to Congress, invited by Rep Barton, Chair of the House Energy and Commerce Committee. The report has recently been revealed as heavily plagiarised. It was the centerpiece of hearings directed at Michael Mann's "hockey-stick" papers (MBH98, Nature 1998,MBH99)

However, this post is about the science. The thrust of the WR scientific criticism of MBH is that they used an inappropriate mean to normalize the proxy data - the mean for the calibration period, rather than the full period. This would tend to produce hockey-stick results.

The WR report was based on papers by McIntyre and McKitrick, particularly MM05b GRL. Wegman used their code, archived here. An important claim, frequently cited, is that the MBH algorithm would generate results of hockey-stick appearance, even if the data consisted of red noise with no such tendency. To this end, they showed three figures based on red noise simulations:
  • Fig 4.1 compared the first PC generated from such a simulation with the MBH reconstruction.
  • Fig 4.2 showed a histogram of "hockey-stick index" (a difference of means as a measure of HS shape)for 10,000 simulations using the limited and the full mean.It showed a normal unimodal distribution for the full mean ("centered"), and a bimodal distribution for the partial mean ("decentered").
  • Fig 4.4 came with this caption:
    One of the most compelling illustrations that McIntyre and McKitrick have produced is created by feeding red noise [AR(1) with parameter = 0.2] into the MBH algorithm. The AR(1) process is a stationary process meaning that it should not exhibit any long-term trend. The MBH98 algorithm found ‘hockey stick’ trend in each of the independent replications.
    It showed twelve HS-like PC1's generated from a MBH algorithm.

Deep Climate did a thorough investigation of these graphs and their provenance, to complement the work he and John Mashey did on the plagiarism. Regarding these plots he found:
  • the HS PC's shown were anything but random samples. In fact, the 10000 simulations had been pre-sorted by HS index, and the top 100 selected. A choice was then made from this top 100.
  • Although Wegman had said that "We have been able to reproduce the results of McIntyre and McKitrick (2005b)", the PC in Fig 4.1 was identical to one in MM05b. Since the noise is randomly generated, this could not have happened from a proper re-run of the code. Somehow, the graph was produced from MM05 computed results.
  • The red noise used in the program was very different to that described in the caption of Fig 4.4.

In this post, I mainly want to concentrate on the first issue. How much of the HS shape of the PC's that they showed was due to the MBH selection process (and there is some), and how much to the artificial selection from the top 1% of sorted HS shapes? To this end, I tried running the same algorithm with the same red noise, but using correct centering.

It's a fairly long post, but you can peek at the conclusion.


This post arises partly from a thread at Climate Audit. A commenter, oneuniverse, undertook the task of re-running the MM05 code. You can find his comments near here. His results are here.

I should first point out that Fig 4.2 is not affected by the selection, and oneuniverse correctly points out that his simulations, which do not make the HS index selection, return essentially the same results. He also argues that these are the most informative, which may well be true, although the thing plotted, HS index, is not intuitive. It was the HS-like profiles in Figs 4.1 and 4.4 that attracted attention.

However, it is also clear that the selection process did make the plots in Figs 4.1 and 4.4 more HS-like.

My own results may differ slightly from those of oneuniverse, in that I took the view that the MM05 code was mixing multiple issues. They noted that MBH also standardised twice by dividing by sd. So they did an explicit svd calc for the MBH emulations, but a standard R prcomp for the centered emulations. I took the view that since it is the decentering that is being studied, it is better to compare the effect of changing just that. I don't believe the other differences matter much, but I think it is better practice to vary one thing at a time.

I'll focus on Fig 4.4, since the PC shown in Fig 4.1 might as well be taken from that selection. Here is the original from the Wegman Report:


And here is my corresponding emulation, using the same selection procedures. It isn't identical to the WR version, but shows the same features.


Now here is what you get if you use centered differencing in the same program. I did this by replacing the calibration mean by the full sample mean but leaving everything else (there's a bit more to it - see update below).


Clearly, there is also a strong appearance of HS shape. But this has nothing to do with the decentered mean. It is the result of the prior selection for HS shape that Wegman used.
You'll notice that the scaling is also somewhat different. This is due to MBH dividing twice (in effect) by the sd in normalising, at least in the MM05 version. I'm not sure that this is a good idea, but it shouldn't make much difference in PCA. The second denominator in the MBH case is larger, because it is calculated relative to the deviant mean. Incidentally, the scaling in the original MM05 code was very different again.

And here is a properly representative sample of the decentered PC's. I simply took a consecutive block (actually 9001-9100) and made the same selection from those, instead of the sorted subsample. I didn't change the numbers within the 100 - because the data is randomly generated, it should be possible to then choose a subsample arbitrarily, rather than regenerate a random selection.


Now we see that there is still some tendency to HS shape, but much less. It can go either way, as expected. In the PCA analysis, sign doesn't matter, so the sign variations don't cancel.

Finally, here is the corresponding randomly chosen centered version. There is essentially no HS tendency.


The last two plots are a fairer indication of the HS tendency than is seen in Fig 4.4 of the Wegman report. It isn't nothing, but it isn't as neat as portrayed there.

Update: I should clarify "leaving everything else" . The "mannomatic" transform in MM05 is:
  mannomatic = function(x) {N = length(x); i=(N-MK):N;  xstd = (x- mean( x[i]))/sd(x[i]);
    sdprox = sd.detrend(xstd[i]); mannomatic = xstd/sdprox; mannomatic }

    I've modified the MM05 version for clarity. MK was set to 78, the number of years in the calibration region. It determined the range i, which is used both for mean and standard deviation, and the further normalisation. My "centered" version sets MK=N-1 (all years) rather than 78.


Update: Conclusion.
  1. Wegman, using the code of MM05b, claimed that the technique of MBH (decentering) would yield hockey-stick shaped PC1's even from red noise input (1st fig).
  2. The 2nd fig confirms this. However, the effect is part due to MBH, and part due to a very artificial selection in the MM05 code, where a subsample (100 from 10000) was selected for HS shape prior to display.
  3. The third fig shows that this artificial selection will itself create HS shapes without decentering - no MBH effect
  4. The fourth fig shows how Wegman's Fig 4.4 should have looked, without the artificial selection. Some HS effect, but not nearly as much.
  5. The final fig just confirms that with no selection and no decentering, the HS goes away.

Here is a summary of the plots, for easier comparison:

             Like Fig 4.4 - decentered, selected

                centered (MM05 alg - see appendix), and selected

                     decentered, not selected

            centered, not selected

Update. I have uploaded the R files - see MM05_NS.zip

 
Update - Appendix. 

Following the suggestion of oneuniverse, I have recalculated the effect of selection with a centered algorithm, done according to the original MM05b algorithm instead of my adaption of the MBH algorithm. It corresponds to the third plot above. It is of course from a different red noise instance, since the program was re-run. The prcomm() algorithm returns a very different scaling.

It appears to be to also have a very strong HS appearance - but judge for yourself. Again, I emphasise that this is done with the centered MM05 algorithm, as used by Wegman, and the HS derives sinmply from the artificial selection, not from anything unusual in MBH.