Overview

The CausalImpact library measures the effects of an event on a response variable when establishing a traditional control group through a randomized trial is not a viable option. It does this by establishing a ‘synthetic control’ which serves as a baseline under which the actual data is compared.

In this tutorial, we’ll look at the effect that the Corona virus outbreak had on the number of “Make an Appointment” forms completed on a hospital website. To begin, we must establish a “pre-period” before the event occurred and a “post-period” after the event occurred. The pre-period is used to train a Bayesian Structural Time Series model. In the post-period, the model is used to predict our synthetic control which indicates how the outcome may have performed were the event not to have occurred.

Our pre-period will be 10/1/2019 to 3/15/2020 and our post-period will be 3/16/2020 - 5/4/2020. Our predictor variables will be the number of sessions from organic, social, and referral sources. An important assumption made by the CausalImpact library is that our predictors are not affected by our event.

First, we must gather the data necessary for our analysis. Our response variable, as established earlier, will be “Make an Appointment” form completions which is the goal1Completions metric in GA. Our predictor variables will come from the channelGrouping dimension in GA.

We know that the hospital suspended paid media around the time of the outbreak so we’ll remove traffic from paid sources using the following filter:

channel_filter <- dim_filter(dimension="channelGrouping",operator="REGEXP",expressions="Paid Search|Display",not = T)

We call the Google Analytics reporting API twice. Once to gather the goal completion data:

# Gather goal data
date_range = date_range,
metrics = "goal1Completions",
dimensions = c("date"),
dim_filters = my_filter_clause,
max = -1)

and once to gather the channel session data:

df_sessions <- google_analytics(viewId = view_id,
date_range = date_range,
metrics = c("sessions"),
dimensions = c("date","channelGrouping"),
max = -1,
dim_filters = my_filter_clause)

This avoids us having to aggregate the goal data after pivoting the session data. Pivoting the session data generates multiple columns of data from our single channelGrouping column. Putting this all together is shown below.


date_range <- c("2019-10-01","2020-05-04")

# Remove paid traffic
channel_filter <- dim_filter(dimension="channelGrouping",operator="REGEXP",expressions="Paid Search|Display",not = T)
my_filter_clause <- filter_clause_ga4(list(channel_filter))

# Gather goal data
date_range = date_range,
metrics = "goal1Completions",
dimensions = c("date"),
dim_filters = my_filter_clause,
max = -1)
# Gather session data
date_range = date_range,
metrics = c("sessions"),
dimensions = c("date","channelGrouping"),
max = -1,
dim_filters = my_filter_clause) %>%
pivot_wider(id_cols=date,names_from=channelGrouping,values_from=sessions) %>%
mutate_at(vars(-date),~if_else(is.na(.),0,.))

# Merge the goal completion data into the sessions data
df <- df_sessions %>% mutate(y = df_goals$goal1Completions)   head(df) %>% gt() date (Other) Direct Email Organic Search Referral Social y 2019-10-01 6 21063 1938 13216 2131 423 26 2019-10-02 8 20247 524 13190 2180 397 26 2019-10-03 3 20314 323 12610 2051 410 19 2019-10-04 6 18417 774 10959 1794 785 12 2019-10-05 5 5822 170 4793 490 727 10 2019-10-06 5 5347 84 5269 500 714 16 NA Create BSTS Model The following code creates a Bayesian Structural Time Series model that will be used by the CausalImpact library to generate our synthetic control. It’s here that we input our pre-period and post-period as well as our predictor and response variables. The BSTS package has several options for modifying our model. Here, we apply a “local level” which captures high level trend in the response variable. We also capture the 7-day weekly trend in our data using AddSeasonal(). df2 <- df # Create copy of our DF so we can re-run after the remove the response data from prediction period # Assign pre and post periods pre.period <- c(1,which(df$date == "2020-03-15"))
post.period <- c(which(df$date == "2020-03-15")+1,length(df$date))
post.period.response <- df$y[post.period[1] : post.period[2]] # Remove outcomes from the post-period. The BSTS model should be ignorant of the values we intend to predict df2$y[post.period[1] : post.period[2]] <- NA

# Create a zoo object which adds dates to plot output
df_zoo <- read.zoo(df2, format = "%Y-%m-%d")

# Add loacl and seasonal trends
ss <- AddLocalLevel(list(), df_zoo$y) ss <- AddSeasonal(ss, df_zoo$y, nseasons = 7) # weekly seasonal trend
bsts.model <- bsts(y ~ ., ss, niter = 1000, data = df3_zoo, family = "gaussian", ping=0)

plot(bsts.model)


The blue dots are the actual data points and the black line underneath is our estimated posterior distribution. We can see that the model does a reasonable job of predicting form completions, though there are some outliers in late February that are not well predicted. This will increase our uncertainty in our predictions and thus widen our confidence interval (the shading around the black line).

Generate CausalImpact Analysis

Now that we have our model, we can compare our prediction to what actually happened and measure the impact of the event.

impact <- CausalImpact(bsts.model = bsts.model,
post.period.response = post.period.response)

plot(impact)

The top plot shows the actual data in black and our predicted distribution of the response variable in blue with the median value as a dashed blue line. The 2nd plot subtracts the predicted data from the actual data to show the difference between the two values. If th effect had no impact, we would expect the pointwise estimated to hover around 0. The last plot shows the cumulative impact of the event over time. Notice how our confidence interval (shown in blue) widens as time goes on.

Our causal impact model confirms a decrease in the number of form completions, however the 95% confidence interval quickly includes 0 which means that we cannot say with certainty that the impact extends into April. While we weren’t able to find conclusive results, being able to measure our certainty is a major benefit of Bayesian models such as this one.

Causal Impact Report

One nice feature of the CausalImpact library is that it provides a human-friendly read-out of the results. Here they are summarized below.

 summary(impact, "report")
Analysis report {CausalImpact}

During the post-intervention period, the response variable had an average value of approx. 8.26. In the absence of an intervention, we would have expected an average response of 13.40. The 95% interval of this counterfactual prediction is [2.20, 22.40]. Subtracting this prediction from the observed response yields an estimate of the causal effect the intervention had on the response variable. This effect is -5.14 with a 95% interval of [-14.14, 6.06]. For a discussion of the significance of this effect, see below.

Summing up the individual data points during the post-intervention period (which can only sometimes be meaningfully interpreted), the response variable had an overall value of 413.00. Had the intervention not taken place, we would have expected a sum of 669.75. The 95% interval of this prediction is [110.12, 1120.16].

The above results are given in terms of absolute numbers. In relative terms, the response variable showed a decrease of -38%. The 95% interval of this percentage is [-106%, +45%].

This means that, although it may look as though the intervention has exerted a negative effect on the response variable when considering the intervention period as a whole, this effect is not statistically significant, and so cannot be meaningfully interpreted. The apparent effect could be the result of random fluctuations that are unrelated to the intervention. This is often the case when the intervention period is very long and includes much of the time when the effect has already worn off. It can also be the case when the intervention period is too short to distinguish the signal from the noise. Finally, failing to find a significant effect can happen when there are not enough control variables or when these variables do not correlate well with the response variable during the learning period.

The probability of obtaining this effect by chance is p = 0.144. This means the effect may be spurious and would generally not be considered statistically significant. 

Validating Our Synthetic Control

One method of validating your model is to generate predictions before the event occurred. If our model is well-behaved, we should see little difference between the predicted and actual response data.


# Filter to include only pre-event data. Also reorder columns to place y after the date
df_compare <- df %>% filter(date < "2020-02-15") %>% select(date,last_col(),2:length(df))

df_zoo <- read.zoo(df_compare, format = "%Y-%m-%d")

pre.period <- c(index(df_zoo)[1],index(df_zoo)[which(df_compare$date == "2020-01-15")]) post.period <- c(index(df_zoo)[which(df_compare$date == "2020-01-15")+1],index(df_zoo)[length(df_compare\$date)])

impact <- CausalImpact(df_zoo, pre.period, post.period)

plot(impact)

Above we see that the model doesn’t do a great job of predicting the upper spikes of the form completions which likely explains the wide confidence interval seen earlier.

Comparison to the Naive Approach

Deploying advanced modeling techniques is only useful if there are advantages over much simpler techniques. The naive method would be to use our pre-intervention data to establish an average and continue that average into the post-period to estimate a synthetic control.

Before the event, we had about 19 form fills a day. After, we had 8.5 a day. That’s a decrease of about 52%. CausalImpact estimated a decrease in 44% with a 95% confidence interval of 29%-63%. Were these numbers to be substantially different, and we had confidence in our model, we would prefer the figures generated by CausalImpact.

There are some clear cases when modeling will outperform the naive approach described above:

1: If there is a trend in the response variable, then averaging the pre-period will not capture the continuation of that trend. 2. If evaluating the degree of confidence is important, the CausalImpact model is preferable due to its ability to measure uncertainty.

LS0tDQp0aXRsZTogIk1lYXN1cmluZyBDYXVzYWwgSW1wYWN0IHdpdGggR0EgRGF0YSAtIEV2YWx1YXRpbmcgdGhlIEVmZmVjdHMgb2YgQ09WSUQxOSBvbiBIb3NwaXRhbCBBcHBvaW50bWVudHMiDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KDQpgYGB7ciBzZXR1cCwgbWVzc2FnZT1GQUxTRSx3YXJuaW5nPUZBTFNFLGVjaG89RkFMU0V9DQoNCmxpYnJhcnkoQ2F1c2FsSW1wYWN0KQ0KbGlicmFyeShnb29nbGVBbmFseXRpY3NSKQ0KbGlicmFyeShnb29nbGVBdXRoUikNCmxpYnJhcnkodGlkeXZlcnNlKQ0KbGlicmFyeSh6b28pDQpsaWJyYXJ5KGJzdHMpDQpsaWJyYXJ5KGd0KQ0KZ2FyX2F1dGgoZW1haWwgPSBTeXMuZ2V0ZW52KCJPQVVUSF9FTUFJTCIpKSAjIEFjdHVhbCBlbWFpbCBoaWRkZW4gdG8gcHJvdGVjdCB0aGUgYWNjb3VudA0Kdmlld19pZCA8LSBTeXMuZ2V0ZW52KCJDQVVTQUxfSU5GX1ZJRVdfSUQiKSAjIEFjdHVhbCBHQSB2aWV3IElEIGhpZGRlbiB0byBwcm90ZWN0IHRoZSBhY2NvdW50DQoNCiMjIyBSZWZlcmVuY2VzDQojIEJTVFMgaHR0cDovL3d3dy51bm9mZmljaWFsZ29vZ2xlZGF0YXNjaWVuY2UuY29tLzIwMTcvMDcvZml0dGluZy1iYXllc2lhbi1zdHJ1Y3R1cmFsLXRpbWUtc2VyaWVzLmh0bWwNCiMgQ2F1c2FsSW1wYWN0IGh0dHBzOi8vZ29vZ2xlLmdpdGh1Yi5pby9DYXVzYWxJbXBhY3QvQ2F1c2FsSW1wYWN0Lmh0bWwNCmBgYA0KDQojIE92ZXJ2aWV3DQoNClRoZSBbQ2F1c2FsSW1wYWN0XShodHRwczovL2dvb2dsZS5naXRodWIuaW8vQ2F1c2FsSW1wYWN0L0NhdXNhbEltcGFjdC5odG1sKSBsaWJyYXJ5IG1lYXN1cmVzIHRoZSBlZmZlY3RzIG9mIGFuIGV2ZW50IG9uIGEgcmVzcG9uc2UgdmFyaWFibGUgd2hlbiBlc3RhYmxpc2hpbmcgYSB0cmFkaXRpb25hbCBjb250cm9sIGdyb3VwIHRocm91Z2ggYSByYW5kb21pemVkIHRyaWFsIGlzIG5vdCBhIHZpYWJsZSBvcHRpb24uIEl0IGRvZXMgdGhpcyBieSBlc3RhYmxpc2hpbmcgYSAnc3ludGhldGljIGNvbnRyb2wnIHdoaWNoIHNlcnZlcyBhcyBhIGJhc2VsaW5lIHVuZGVyIHdoaWNoIHRoZSBhY3R1YWwgZGF0YSBpcyBjb21wYXJlZC4gDQoNCkluIHRoaXMgdHV0b3JpYWwsIHdlJ2xsIGxvb2sgYXQgdGhlIGVmZmVjdCB0aGF0IHRoZSBDb3JvbmEgdmlydXMgb3V0YnJlYWsgaGFkIG9uIHRoZSBudW1iZXIgb2YgIk1ha2UgYW4gQXBwb2ludG1lbnQiIGZvcm1zIGNvbXBsZXRlZCBvbiBhIGhvc3BpdGFsIHdlYnNpdGUuIFRvIGJlZ2luLCB3ZSBtdXN0IGVzdGFibGlzaCBhICJwcmUtcGVyaW9kIiBiZWZvcmUgdGhlIGV2ZW50IG9jY3VycmVkIGFuZCBhICJwb3N0LXBlcmlvZCIgYWZ0ZXIgdGhlIGV2ZW50IG9jY3VycmVkLiBUaGUgcHJlLXBlcmlvZCBpcyB1c2VkIHRvIHRyYWluIGEgQmF5ZXNpYW4gU3RydWN0dXJhbCBUaW1lIFNlcmllcyBtb2RlbC4gSW4gdGhlIHBvc3QtcGVyaW9kLCB0aGUgbW9kZWwgaXMgdXNlZCB0byBwcmVkaWN0IG91ciBzeW50aGV0aWMgY29udHJvbCB3aGljaCBpbmRpY2F0ZXMgaG93IHRoZSBvdXRjb21lIG1heSBoYXZlIHBlcmZvcm1lZCB3ZXJlIHRoZSBldmVudCBub3QgdG8gaGF2ZSBvY2N1cnJlZC4NCg0KT3VyIHByZS1wZXJpb2Qgd2lsbCBiZSAxMC8xLzIwMTkgdG8gMy8xNS8yMDIwIGFuZCBvdXIgcG9zdC1wZXJpb2Qgd2lsbCBiZSAzLzE2LzIwMjAgLSA1LzQvMjAyMC4gT3VyIHByZWRpY3RvciB2YXJpYWJsZXMgd2lsbCBiZSB0aGUgbnVtYmVyIG9mIHNlc3Npb25zIGZyb20gb3JnYW5pYywgc29jaWFsLCBhbmQgcmVmZXJyYWwgc291cmNlcy4gQW4gaW1wb3J0YW50IGFzc3VtcHRpb24gbWFkZSBieSB0aGUgQ2F1c2FsSW1wYWN0IGxpYnJhcnkgaXMgdGhhdCBvdXIgcHJlZGljdG9ycyBhcmUgKm5vdCogYWZmZWN0ZWQgYnkgb3VyIGV2ZW50Lg0KDQojIEdhdGhlcmluZyBEYXRhIGZyb20gR29vZ2xlIEFuYWx5dGljcw0KDQpGaXJzdCwgd2UgbXVzdCBnYXRoZXIgdGhlIGRhdGEgbmVjZXNzYXJ5IGZvciBvdXIgYW5hbHlzaXMuIE91ciByZXNwb25zZSB2YXJpYWJsZSwgYXMgZXN0YWJsaXNoZWQgZWFybGllciwgd2lsbCBiZSAiTWFrZSBhbiBBcHBvaW50bWVudCIgZm9ybSBjb21wbGV0aW9ucyB3aGljaCBpcyB0aGUgZ29hbDFDb21wbGV0aW9ucyBtZXRyaWMgaW4gR0EuIE91ciBwcmVkaWN0b3IgdmFyaWFibGVzIHdpbGwgY29tZSBmcm9tIHRoZSBjaGFubmVsR3JvdXBpbmcgZGltZW5zaW9uIGluIEdBLg0KDQpXZSBrbm93IHRoYXQgdGhlIGhvc3BpdGFsIHN1c3BlbmRlZCBwYWlkIG1lZGlhIGFyb3VuZCB0aGUgdGltZSBvZiB0aGUgb3V0YnJlYWsgc28gd2UnbGwgcmVtb3ZlIHRyYWZmaWMgZnJvbSBwYWlkIHNvdXJjZXMgdXNpbmcgdGhlIGZvbGxvd2luZyBmaWx0ZXI6DQoNCmBgYHtyIGV2YWw9IEZBTFNFfQ0KY2hhbm5lbF9maWx0ZXIgPC0gZGltX2ZpbHRlcihkaW1lbnNpb249ImNoYW5uZWxHcm91cGluZyIsb3BlcmF0b3I9IlJFR0VYUCIsZXhwcmVzc2lvbnM9IlBhaWQgU2VhcmNofERpc3BsYXkiLG5vdCA9IFQpDQpgYGANCg0KV2UgY2FsbCB0aGUgR29vZ2xlIEFuYWx5dGljcyByZXBvcnRpbmcgQVBJIHR3aWNlLiBPbmNlIHRvIGdhdGhlciB0aGUgZ29hbCBjb21wbGV0aW9uIGRhdGE6DQoNCmBgYHtyIGV2YWw9RkFMU0V9DQojIEdhdGhlciBnb2FsIGRhdGENCmRmX2dvYWxzIDwtIGdvb2dsZV9hbmFseXRpY3Modmlld0lkID0gdmlld19pZCwNCiAgICAgICAgICAgICAgICAgICAgICAgZGF0ZV9yYW5nZSA9IGRhdGVfcmFuZ2UsDQogICAgICAgICAgICAgICAgICAgICAgIG1ldHJpY3MgPSAiZ29hbDFDb21wbGV0aW9ucyIsDQogICAgICAgICAgICAgICAgICAgICAgIGRpbWVuc2lvbnMgPSBjKCJkYXRlIiksDQogICAgICAgICAgICAgICAgICAgICAgIGRpbV9maWx0ZXJzID0gbXlfZmlsdGVyX2NsYXVzZSwNCiAgICAgICAgICAgICAgICAgICAgICAgbWF4ID0gLTEpDQpgYGANCiANCmFuZCBvbmNlIHRvIGdhdGhlciB0aGUgY2hhbm5lbCBzZXNzaW9uIGRhdGE6DQoNCmBgYHtyIGV2YWw9RkFMU0V9DQpkZl9zZXNzaW9ucyA8LSBnb29nbGVfYW5hbHl0aWNzKHZpZXdJZCA9IHZpZXdfaWQsDQogICAgICAgICAgICAgICAgICAgICAgIGRhdGVfcmFuZ2UgPSBkYXRlX3JhbmdlLA0KICAgICAgICAgICAgICAgICAgICAgICBtZXRyaWNzID0gYygic2Vzc2lvbnMiKSwNCiAgICAgICAgICAgICAgICAgICAgICAgZGltZW5zaW9ucyA9IGMoImRhdGUiLCJjaGFubmVsR3JvdXBpbmciKSwNCiAgICAgICAgICAgICAgICAgICAgICAgbWF4ID0gLTEsDQogICAgICAgICAgICAgICAgICAgICAgIGRpbV9maWx0ZXJzID0gbXlfZmlsdGVyX2NsYXVzZSkNCmBgYA0KDQpUaGlzIGF2b2lkcyB1cyBoYXZpbmcgdG8gYWdncmVnYXRlIHRoZSBnb2FsIGRhdGEgYWZ0ZXIgcGl2b3RpbmcgdGhlIHNlc3Npb24gZGF0YS4gUGl2b3RpbmcgdGhlIHNlc3Npb24gZGF0YSBnZW5lcmF0ZXMgbXVsdGlwbGUgY29sdW1ucyBvZiBkYXRhIGZyb20gb3VyIHNpbmdsZSBjaGFubmVsR3JvdXBpbmcgY29sdW1uLiBQdXR0aW5nIHRoaXMgYWxsIHRvZ2V0aGVyIGlzIHNob3duIGJlbG93Lg0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KDQpkYXRlX3JhbmdlIDwtIGMoIjIwMTktMTAtMDEiLCIyMDIwLTA1LTA0IikNCg0KIyBSZW1vdmUgcGFpZCB0cmFmZmljDQpjaGFubmVsX2ZpbHRlciA8LSBkaW1fZmlsdGVyKGRpbWVuc2lvbj0iY2hhbm5lbEdyb3VwaW5nIixvcGVyYXRvcj0iUkVHRVhQIixleHByZXNzaW9ucz0iUGFpZCBTZWFyY2h8RGlzcGxheSIsbm90ID0gVCkNCm15X2ZpbHRlcl9jbGF1c2UgPC0gZmlsdGVyX2NsYXVzZV9nYTQobGlzdChjaGFubmVsX2ZpbHRlcikpDQoNCiMgR2F0aGVyIGdvYWwgZGF0YQ0KZGZfZ29hbHMgPC0gZ29vZ2xlX2FuYWx5dGljcyh2aWV3SWQgPSB2aWV3X2lkLA0KICAgICAgICAgICAgICAgICAgICAgICBkYXRlX3JhbmdlID0gZGF0ZV9yYW5nZSwNCiAgICAgICAgICAgICAgICAgICAgICAgbWV0cmljcyA9ICJnb2FsMUNvbXBsZXRpb25zIiwNCiAgICAgICAgICAgICAgICAgICAgICAgZGltZW5zaW9ucyA9IGMoImRhdGUiKSwNCiAgICAgICAgICAgICAgICAgICAgICAgZGltX2ZpbHRlcnMgPSBteV9maWx0ZXJfY2xhdXNlLA0KICAgICAgICAgICAgICAgICAgICAgICBtYXggPSAtMSkNCiMgR2F0aGVyIHNlc3Npb24gZGF0YQ0KZGZfc2Vzc2lvbnMgPC0gZ29vZ2xlX2FuYWx5dGljcyh2aWV3SWQgPSB2aWV3X2lkLA0KICAgICAgICAgICAgICAgICAgICAgICBkYXRlX3JhbmdlID0gZGF0ZV9yYW5nZSwNCiAgICAgICAgICAgICAgICAgICAgICAgbWV0cmljcyA9IGMoInNlc3Npb25zIiksDQogICAgICAgICAgICAgICAgICAgICAgIGRpbWVuc2lvbnMgPSBjKCJkYXRlIiwiY2hhbm5lbEdyb3VwaW5nIiksDQogICAgICAgICAgICAgICAgICAgICAgIG1heCA9IC0xLA0KICAgICAgICAgICAgICAgICAgICAgICBkaW1fZmlsdGVycyA9IG15X2ZpbHRlcl9jbGF1c2UpICU+JSANCiAgIHBpdm90X3dpZGVyKGlkX2NvbHM9ZGF0ZSxuYW1lc19mcm9tPWNoYW5uZWxHcm91cGluZyx2YWx1ZXNfZnJvbT1zZXNzaW9ucykgJT4lDQogIG11dGF0ZV9hdCh2YXJzKC1kYXRlKSx+aWZfZWxzZShpcy5uYSguKSwwLC4pKSANCg0KIyBNZXJnZSB0aGUgZ29hbCBjb21wbGV0aW9uIGRhdGEgaW50byB0aGUgc2Vzc2lvbnMgZGF0YQ0KZGYgPC0gZGZfc2Vzc2lvbnMgJT4lIG11dGF0ZSh5ID0gZGZfZ29hbHMkZ29hbDFDb21wbGV0aW9ucykNCg0KYGBgDQoNCmBgYHtyfQ0KDQpoZWFkKGRmKSAlPiUgZ3QoKQ0KDQpgYGANCg0KDQojIENyZWF0ZSBCU1RTIE1vZGVsDQoNClRoZSBmb2xsb3dpbmcgY29kZSBjcmVhdGVzIGEgQmF5ZXNpYW4gU3RydWN0dXJhbCBUaW1lIFNlcmllcyBtb2RlbCB0aGF0IHdpbGwgYmUgdXNlZCBieSB0aGUgQ2F1c2FsSW1wYWN0IGxpYnJhcnkgdG8gZ2VuZXJhdGUgb3VyIHN5bnRoZXRpYyBjb250cm9sLiBJdCdzIGhlcmUgdGhhdCB3ZSBpbnB1dCBvdXIgcHJlLXBlcmlvZCBhbmQgcG9zdC1wZXJpb2QgYXMgd2VsbCBhcyBvdXIgcHJlZGljdG9yIGFuZCByZXNwb25zZSB2YXJpYWJsZXMuIA0KDQpUaGUgQlNUUyBwYWNrYWdlIGhhcyBzZXZlcmFsIG9wdGlvbnMgZm9yIG1vZGlmeWluZyBvdXIgbW9kZWwuIEhlcmUsIHdlIGFwcGx5IGEgImxvY2FsIGxldmVsIiB3aGljaCBjYXB0dXJlcyBoaWdoIGxldmVsIHRyZW5kIGluIHRoZSByZXNwb25zZSB2YXJpYWJsZS4gV2UgYWxzbyBjYXB0dXJlIHRoZSA3LWRheSB3ZWVrbHkgdHJlbmQgaW4gb3VyIGRhdGEgdXNpbmcgYEFkZFNlYXNvbmFsKClgLiANCg0KYGBge3Igd2FybmluZz1GQUxTRSxtZXNzYWdlPUZBTFNFfQ0KZGYyIDwtIGRmICMgQ3JlYXRlIGNvcHkgb2Ygb3VyIERGIHNvIHdlIGNhbiByZS1ydW4gYWZ0ZXIgdGhlIHJlbW92ZSB0aGUgcmVzcG9uc2UgZGF0YSBmcm9tIHByZWRpY3Rpb24gcGVyaW9kDQoNCiMgQXNzaWduIHByZSBhbmQgcG9zdCBwZXJpb2RzDQpwcmUucGVyaW9kIDwtIGMoMSx3aGljaChkZiRkYXRlID09ICIyMDIwLTAzLTE1IikpDQpwb3N0LnBlcmlvZCA8LSBjKHdoaWNoKGRmJGRhdGUgPT0gIjIwMjAtMDMtMTUiKSsxLGxlbmd0aChkZiRkYXRlKSkNCnBvc3QucGVyaW9kLnJlc3BvbnNlIDwtIGRmJHlbcG9zdC5wZXJpb2RbMV0gOiBwb3N0LnBlcmlvZFsyXV0NCg0KIyBSZW1vdmUgb3V0Y29tZXMgZnJvbSB0aGUgcG9zdC1wZXJpb2QuIFRoZSBCU1RTIG1vZGVsIHNob3VsZCBiZSBpZ25vcmFudCBvZiB0aGUgdmFsdWVzIHdlIGludGVuZCB0byBwcmVkaWN0DQpkZjIkeVtwb3N0LnBlcmlvZFsxXSA6IHBvc3QucGVyaW9kWzJdXSA8LSBOQQ0KDQojIENyZWF0ZSBhIHpvbyBvYmplY3Qgd2hpY2ggYWRkcyBkYXRlcyB0byBwbG90IG91dHB1dA0KZGZfem9vIDwtIHJlYWQuem9vKGRmMiwgZm9ybWF0ID0gIiVZLSVtLSVkIikgDQoNCiMgQWRkIGxvYWNsIGFuZCBzZWFzb25hbCB0cmVuZHMNCnNzIDwtIEFkZExvY2FsTGV2ZWwobGlzdCgpLCBkZl96b28keSkNCnNzIDwtIEFkZFNlYXNvbmFsKHNzLCBkZl96b28keSwgbnNlYXNvbnMgPSA3KSAjIHdlZWtseSBzZWFzb25hbCB0cmVuZA0KYnN0cy5tb2RlbCA8LSBic3RzKHkgfiAuLCBzcywgbml0ZXIgPSAxMDAwLCBkYXRhID0gZGYzX3pvbywgZmFtaWx5ID0gImdhdXNzaWFuIiwgcGluZz0wKQ0KDQpwbG90KGJzdHMubW9kZWwpDQoNCmBgYA0KDQpUaGUgYmx1ZSBkb3RzIGFyZSB0aGUgYWN0dWFsIGRhdGEgcG9pbnRzIGFuZCB0aGUgYmxhY2sgbGluZSB1bmRlcm5lYXRoIGlzIG91ciBlc3RpbWF0ZWQgcG9zdGVyaW9yIGRpc3RyaWJ1dGlvbi4gV2UgY2FuIHNlZSB0aGF0IHRoZSBtb2RlbCBkb2VzIGEgcmVhc29uYWJsZSBqb2Igb2YgcHJlZGljdGluZyBmb3JtIGNvbXBsZXRpb25zLCB0aG91Z2ggdGhlcmUgYXJlIHNvbWUgb3V0bGllcnMgaW4gbGF0ZSBGZWJydWFyeSB0aGF0IGFyZSBub3Qgd2VsbCBwcmVkaWN0ZWQuICBUaGlzIHdpbGwgaW5jcmVhc2Ugb3VyIHVuY2VydGFpbnR5IGluIG91ciBwcmVkaWN0aW9ucyBhbmQgdGh1cyB3aWRlbiBvdXIgY29uZmlkZW5jZSBpbnRlcnZhbCAodGhlIHNoYWRpbmcgYXJvdW5kIHRoZSBibGFjayBsaW5lKS4NCg0KIyBHZW5lcmF0ZSBDYXVzYWxJbXBhY3QgQW5hbHlzaXMNCg0KTm93IHRoYXQgd2UgaGF2ZSBvdXIgbW9kZWwsIHdlIGNhbiBjb21wYXJlIG91ciBwcmVkaWN0aW9uIHRvIHdoYXQgYWN0dWFsbHkgaGFwcGVuZWQgYW5kIG1lYXN1cmUgdGhlIGltcGFjdCBvZiB0aGUgZXZlbnQuIA0KDQpgYGB7cn0NCmltcGFjdCA8LSBDYXVzYWxJbXBhY3QoYnN0cy5tb2RlbCA9IGJzdHMubW9kZWwsDQogICAgICAgICAgICAgICAgICAgICAgIHBvc3QucGVyaW9kLnJlc3BvbnNlID0gcG9zdC5wZXJpb2QucmVzcG9uc2UpDQoNCnBsb3QoaW1wYWN0KQ0KYGBgDQoNClRoZSB0b3AgcGxvdCBzaG93cyB0aGUgYWN0dWFsIGRhdGEgaW4gYmxhY2sgYW5kIG91ciBwcmVkaWN0ZWQgZGlzdHJpYnV0aW9uIG9mIHRoZSByZXNwb25zZSB2YXJpYWJsZSBpbiBibHVlIHdpdGggdGhlIG1lZGlhbiB2YWx1ZSBhcyBhIGRhc2hlZCBibHVlIGxpbmUuIFRoZSAybmQgcGxvdCBzdWJ0cmFjdHMgdGhlIHByZWRpY3RlZCBkYXRhIGZyb20gdGhlIGFjdHVhbCBkYXRhIHRvIHNob3cgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiB0aGUgdHdvIHZhbHVlcy4gSWYgdGggZWZmZWN0IGhhZCBubyBpbXBhY3QsIHdlIHdvdWxkIGV4cGVjdCB0aGUgcG9pbnR3aXNlIGVzdGltYXRlZCB0byBob3ZlciBhcm91bmQgMC4gVGhlIGxhc3QgcGxvdCBzaG93cyB0aGUgY3VtdWxhdGl2ZSBpbXBhY3Qgb2YgdGhlIGV2ZW50IG92ZXIgdGltZS4gTm90aWNlIGhvdyBvdXIgY29uZmlkZW5jZSBpbnRlcnZhbCAoc2hvd24gaW4gYmx1ZSkgd2lkZW5zIGFzIHRpbWUgZ29lcyBvbi4NCg0KT3VyIGNhdXNhbCBpbXBhY3QgbW9kZWwgY29uZmlybXMgYSBkZWNyZWFzZSBpbiB0aGUgbnVtYmVyIG9mIGZvcm0gY29tcGxldGlvbnMsIGhvd2V2ZXIgdGhlIDk1JSBjb25maWRlbmNlIGludGVydmFsIHF1aWNrbHkgaW5jbHVkZXMgMCB3aGljaCBtZWFucyB0aGF0IHdlIGNhbm5vdCBzYXkgd2l0aCBjZXJ0YWludHkgdGhhdCB0aGUgaW1wYWN0IGV4dGVuZHMgaW50byBBcHJpbC4gDQpXaGlsZSB3ZSB3ZXJlbid0IGFibGUgdG8gZmluZCBjb25jbHVzaXZlIHJlc3VsdHMsIGJlaW5nIGFibGUgdG8gbWVhc3VyZSBvdXIgY2VydGFpbnR5IGlzIGEgbWFqb3IgYmVuZWZpdCBvZiBCYXllc2lhbiBtb2RlbHMgc3VjaCBhcyB0aGlzIG9uZS4gDQoNCiMgQ2F1c2FsIEltcGFjdCBSZXBvcnQNCg0KT25lIG5pY2UgZmVhdHVyZSBvZiB0aGUgQ2F1c2FsSW1wYWN0IGxpYnJhcnkgaXMgdGhhdCBpdCBwcm92aWRlcyBhIGh1bWFuLWZyaWVuZGx5IHJlYWQtb3V0IG9mIHRoZSByZXN1bHRzLiBIZXJlIHRoZXkgYXJlIHN1bW1hcml6ZWQgYmVsb3cuDQoNCmBgYHtyfQ0KIHN1bW1hcnkoaW1wYWN0LCAicmVwb3J0IikNCmBgYA0KDQojIFZhbGlkYXRpbmcgT3VyIFN5bnRoZXRpYyBDb250cm9sDQoNCk9uZSBtZXRob2Qgb2YgdmFsaWRhdGluZyB5b3VyIG1vZGVsIGlzIHRvIGdlbmVyYXRlIHByZWRpY3Rpb25zICpiZWZvcmUqIHRoZSBldmVudCBvY2N1cnJlZC4gSWYgb3VyIG1vZGVsIGlzIHdlbGwtYmVoYXZlZCwgd2Ugc2hvdWxkIHNlZSBsaXR0bGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBwcmVkaWN0ZWQgYW5kIGFjdHVhbCByZXNwb25zZSBkYXRhLg0KDQpgYGB7cn0NCg0KIyBGaWx0ZXIgdG8gaW5jbHVkZSBvbmx5IHByZS1ldmVudCBkYXRhLiBBbHNvIHJlb3JkZXIgY29sdW1ucyB0byBwbGFjZSB5IGFmdGVyIHRoZSBkYXRlDQpkZl9jb21wYXJlIDwtIGRmICU+JSBmaWx0ZXIoZGF0ZSA8ICIyMDIwLTAyLTE1IikgJT4lIHNlbGVjdChkYXRlLGxhc3RfY29sKCksMjpsZW5ndGgoZGYpKQ0KDQpkZl96b28gPC0gcmVhZC56b28oZGZfY29tcGFyZSwgZm9ybWF0ID0gIiVZLSVtLSVkIikNCg0KcHJlLnBlcmlvZCA8LSBjKGluZGV4KGRmX3pvbylbMV0saW5kZXgoZGZfem9vKVt3aGljaChkZl9jb21wYXJlJGRhdGUgPT0gIjIwMjAtMDEtMTUiKV0pDQpwb3N0LnBlcmlvZCA8LSBjKGluZGV4KGRmX3pvbylbd2hpY2goZGZfY29tcGFyZSRkYXRlID09ICIyMDIwLTAxLTE1IikrMV0saW5kZXgoZGZfem9vKVtsZW5ndGgoZGZfY29tcGFyZSRkYXRlKV0pDQoNCmltcGFjdCA8LSBDYXVzYWxJbXBhY3QoZGZfem9vLCBwcmUucGVyaW9kLCBwb3N0LnBlcmlvZCkNCg0KDQpwbG90KGltcGFjdCkNCmBgYA0KDQpBYm92ZSB3ZSBzZWUgdGhhdCB0aGUgbW9kZWwgZG9lc24ndCBkbyBhIGdyZWF0IGpvYiBvZiBwcmVkaWN0aW5nIHRoZSB1cHBlciBzcGlrZXMgb2YgdGhlIGZvcm0gY29tcGxldGlvbnMgd2hpY2ggbGlrZWx5IGV4cGxhaW5zIHRoZSB3aWRlIGNvbmZpZGVuY2UgaW50ZXJ2YWwgc2VlbiBlYXJsaWVyLiANCg0KIyBDb21wYXJpc29uIHRvIHRoZSBOYWl2ZSBBcHByb2FjaA0KDQpEZXBsb3lpbmcgYWR2YW5jZWQgbW9kZWxpbmcgdGVjaG5pcXVlcyBpcyBvbmx5IHVzZWZ1bCBpZiB0aGVyZSBhcmUgYWR2YW50YWdlcyBvdmVyIG11Y2ggc2ltcGxlciB0ZWNobmlxdWVzLiBUaGUgbmFpdmUgbWV0aG9kIHdvdWxkIGJlIHRvIHVzZSBvdXIgcHJlLWludGVydmVudGlvbiBkYXRhIHRvIGVzdGFibGlzaCBhbiBhdmVyYWdlIGFuZCBjb250aW51ZSB0aGF0IGF2ZXJhZ2UgaW50byB0aGUgcG9zdC1wZXJpb2QgdG8gZXN0aW1hdGUgYSBzeW50aGV0aWMgY29udHJvbC4gDQoNCkJlZm9yZSB0aGUgZXZlbnQsIHdlIGhhZCBhYm91dCAxOSBmb3JtIGZpbGxzIGEgZGF5LiBBZnRlciwgd2UgaGFkIDguNSBhIGRheS4gVGhhdCdzIGEgZGVjcmVhc2Ugb2YgYWJvdXQgNTIlLiBDYXVzYWxJbXBhY3QgZXN0aW1hdGVkIGEgZGVjcmVhc2UgaW4gNDQlIHdpdGggYSA5NSUgY29uZmlkZW5jZSBpbnRlcnZhbCBvZiAyOSUtNjMlLiBXZXJlIHRoZXNlIG51bWJlcnMgdG8gYmUgc3Vic3RhbnRpYWxseSBkaWZmZXJlbnQsIGFuZCB3ZSBoYWQgY29uZmlkZW5jZSBpbiBvdXIgbW9kZWwsIHdlIHdvdWxkIHByZWZlciB0aGUgZmlndXJlcyBnZW5lcmF0ZWQgYnkgQ2F1c2FsSW1wYWN0Lg0KDQpUaGVyZSBhcmUgc29tZSBjbGVhciBjYXNlcyB3aGVuIG1vZGVsaW5nIHdpbGwgb3V0cGVyZm9ybSB0aGUgbmFpdmUgYXBwcm9hY2ggZGVzY3JpYmVkIGFib3ZlOg0KDQoxOiBJZiB0aGVyZSBpcyBhIHRyZW5kIGluIHRoZSByZXNwb25zZSB2YXJpYWJsZSwgdGhlbiBhdmVyYWdpbmcgdGhlIHByZS1wZXJpb2Qgd2lsbCBub3QgY2FwdHVyZSB0aGUgY29udGludWF0aW9uIG9mIHRoYXQgdHJlbmQuIA0KMi4gSWYgZXZhbHVhdGluZyB0aGUgZGVncmVlIG9mIGNvbmZpZGVuY2UgaXMgaW1wb3J0YW50LCB0aGUgQ2F1c2FsSW1wYWN0IG1vZGVsIGlzIHByZWZlcmFibGUgZHVlIHRvIGl0cyBhYmlsaXR5IHRvIG1lYXN1cmUgdW5jZXJ0YWludHkuDQoNCg0KDQoNCg0K