Contributor Code of Conduct
diff --git a/dev/ISSUE_TEMPLATE.html b/dev/ISSUE_TEMPLATE.html index 7ef28b10..f09d72aa 100644 --- a/dev/ISSUE_TEMPLATE.html +++ b/dev/ISSUE_TEMPLATE.html @@ -32,11 +32,14 @@- - Basic example + Basic example of using designit: plate layout with two factors
- Using custom shuffle schedule +
- + Batch effects and false positives: a simulation study +
- In-vivo study design @@ -107,7 +110,7 @@
- - Basic example + Basic example of using designit: plate layout with two factors
- Using custom shuffle schedule +
- + Batch effects and false positives: a simulation study +
- In-vivo study design @@ -102,7 +105,7 @@
- - Basic example + Basic example of using designit: plate layout with two factors
- Using custom shuffle schedule +
- + Batch effects and false positives: a simulation study +
- In-vivo study design @@ -106,7 +109,7 @@
- - Basic example + Basic example of using designit: plate layout with two factors
- Using custom shuffle schedule +
- + Batch effects and false positives: a simulation study +
- In-vivo study design @@ -1480,7 +1483,7 @@
- She does not randomize the samples with regard to the plate +layout. The naive layout will put each drug candidate or control (DMSO) +in one column. For simplicity, let us assume that the cells treated with +DMSO are put in column 1, and cells treated with the eleven drug +candidates are put in columns 2 to 12. +
- She randomizes the samples with regard to the plate layout, so that +nearby samples are not necessarily of the same condition. +
- The plate effect is radial, i.e. cells in wells on the edges are +more affected by than cells in wells in the middle of the plate. +
- The plate effect is positive, i.e. cells in edge wells show higher +viability than cells in the middle wells. +
- None of the tested compounds regulate cell viability significantly, +i.e. cells treated with compounds and cell treated with DMSO control +have the same expected value of viability. We simulate the effect of +DMSO and compounds by drawing random samples from a normal +distribution. +
- The true effect of compounds and the plate effect is additive, +i.e. our measurement is the sum of the true effect and the plate +effect. +
NA
diff --git a/dev/LICENSE-text.html b/dev/LICENSE-text.html index f304f9b1..41be8abe 100644 --- a/dev/LICENSE-text.html +++ b/dev/LICENSE-text.html @@ -32,11 +32,14 @@License
diff --git a/dev/LICENSE.html b/dev/LICENSE.html index 0c8ceb34..8bbe651c 100644 --- a/dev/LICENSE.html +++ b/dev/LICENSE.html @@ -32,11 +32,14 @@MIT License
diff --git a/dev/articles/NCS22_talk.html b/dev/articles/NCS22_talk.html index 2f94a34f..6a5103aa 100644 --- a/dev/articles/NCS22_talk.html +++ b/dev/articles/NCS22_talk.html @@ -56,11 +56,14 @@Conclusion
-Site built with pkgdown 2.1.0.
+Site built with pkgdown 2.1.1.
diff --git a/dev/articles/basic_examples.html b/dev/articles/basic_examples.html
index 157abdfc..7dd182d8 100644
--- a/dev/articles/basic_examples.html
+++ b/dev/articles/basic_examples.html
@@ -5,7 +5,7 @@
-Basic example • designit
+Basic example of using designit: plate layout with two factors • designit
@@ -17,7 +17,7 @@
-
+
+
+
+
+
+
+Batch effects and false positives: a simulation study • designit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Batch effects and false positives: a simulation
+study
+ Jitao david
+Zhang
+
+
+ Source: vignettes/false_positives.Rmd
+ false_positives.Rmd
+
+
+
+
+
+In this document, we demonstrate the necessity of a proper experiment
+design with a generative model which we use to simulate data with
+“batch” effects. We show that a proper experiment design helps
+experimentalists and analysts make correct inference about the quantity
+of interest that is robust against randomness.
+#> ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
+#> ✔ dplyr 1.1.4 ✔ readr 2.1.5
+#> ✔ forcats 1.0.0 ✔ stringr 1.5.1
+#> ✔ ggplot2 3.5.1 ✔ tibble 3.2.1
+#> ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
+#> ✔ purrr 1.0.2
+#> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
+#> ✖ dplyr::filter() masks stats::filter()
+#> ✖ dplyr::lag() masks stats::lag()
+#> ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
+
+A simple case study about plate effect: the background
+
+Assume we perform an experiment to test the effect of eleven drug
+candidates under development on cell viability. To do so, we treat cells
+in culture with a fixed concentration of each of the eleven candidates,
+and we treat cells with DMSO (dimethyl sulfoxide) as a vehicle control,
+since the drug candidates are all solved in DMSO solutions.
+To assess the effect with regard to the variability intrinsic to the
+experiment setup, we measure the effect of each drug candidate (and
+DMSO) in eight different batches of cells, which are comparable to each
+other.
+In total, we have 96 samples: 11 drug candidates plus one DMSO
+control, 8 samples each. The samples neatly fit into a 96-well
+microtiter plate with 8 rows, and 12 columns.
+In order to avoid batch effects and to make the operation simple, all
+operations and measurements are done by the same careful operator and
+performed at the same time. The operator has two possibilities:
+
+
+What is the difference between the two variants? Option 2 apparently
+involves more planning and labor than option 1. If manual instead of
+robotic pipetting is involved, option 2 is likely error-prone. So why
+bothering considering the later option?
+Randomization pays off when unwanted variance is large enough so that
+it may distort our estimate of the quantity in which we are interested
+in. In our example, the unwanted variance may come from a plate
+effect: due to variances in temperature, humidity, and evaporation
+between wells in the plate, cells may respond differently to even
+the same treatment. Such plate effects are difficult to
+judge practically because they are not known prior to the experiment,
+unless a calibration study is performed where the cells in a microtiter
+plate are indeed treated with the same condition and measurements are
+performed in order to quantify the plate effect. However, it is simple
+to simulate such plate effects in silico with a
+generative model, and test the effect of randomization.
+For simplicity, we make following further assumptions:
+
+
+
+set.seed(2307111)
+
+conditions <- c("DMSO", sprintf("Compound%02d", 1:11))
+# set up batch container
+bc <- BatchContainer$new(
+ dimensions = list(
+ row = 8, col = 12
+ )
+) |>
+ # assign samples with conditions and true effects
+ assign_in_order(
+ data.frame(
+ SampleIndex = 1:96,
+ Compound = factor(rep(conditions, 8), levels = conditions),
+ trueEffect = rnorm(96, mean = 10, sd = 1)
+ )
+ )
+
+
+Simulating a study in which randomization is not used
+
+First we simulate a study in which randomization is not used. In this
+context, it means that the treatment (controls and compounds in columns)
+and the plate effect are correlated. The following plot visualizes the
+layout of the plate, the true effect, the plate effect, and the
+measurement as a sum of the true effect and the plate effect.
+
+# get observations with batch effect
+get_observations <- function(bc) {
+ bc$get_samples() |>
+ mutate(
+ plateEffect = 0.5 * sqrt((row - 4.5)^2 + (col - 6.5)^2),
+ measurement = trueEffect + plateEffect
+ )
+}
+
+
+
+
+
+row
+ col
+ SampleIndex
+ Compound
+ trueEffect
+ plateEffect
+ measurement
+
+
+
+1
+1
+1
+DMSO
+9.530078
+3.259601
+12.78968
+
+
+1
+2
+2
+Compound01
+10.946764
+2.850439
+13.79720
+
+
+1
+3
+3
+Compound02
+10.111003
+2.474874
+12.58588
+
+
+1
+4
+4
+Compound03
+9.640621
+2.150581
+11.79120
+
+
+1
+5
+5
+Compound04
+9.466205
+1.903943
+11.37015
+
+
+1
+6
+6
+Compound05
+11.049547
+1.767767
+12.81731
+
+
+
+
+
+cowplot::plot_grid(
+ plotlist = list(
+ plot_plate(dat,
+ plate = plate,
+ row = row, column = col, .color = Compound,
+ title = "Layout by treatment"
+ ),
+ plot_plate(dat,
+ plate = plate, row = row, column = col, .color = trueEffect,
+ title = "True effect"
+ ),
+ plot_plate(dat,
+ plate = plate, row = row, column = col, .color = plateEffect,
+ title = "Plate effect"
+ ),
+ plot_plate(dat,
+ plate = plate, row = row, column = col, .color = measurement,
+ title = "Measurement"
+ )
+ ), ncol = 2, nrow = 2
+)
+
+When we perform an one-way ANOVA test with the true effect, the
+F-test suggests that there are no significant differences between the
+treatments (p>0.05).
+
+summary(aov(trueEffect ~ Compound, data = dat))
+#> Df Sum Sq Mean Sq F value Pr(>F)
+#> Compound 11 7.61 0.6917 0.657 0.774
+#> Residuals 84 88.43 1.0528
+However, if we consider the measurement, which sums the true effect
+and the plate effect, the F-test suggests that there are significant
+differences between the compounds (p<0.01).
+
+summary(aov(measurement ~ Compound, data = dat))
+#> Df Sum Sq Mean Sq F value Pr(>F)
+#> Compound 11 41.12 3.738 2.925 0.00258 **
+#> Residuals 84 107.34 1.278
+#> ---
+#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
+To verify, we calculate Turkey’s honest significant differences using
+true effect. As expected, no single compound shows significant
+difference from the effect of DMSO (adjusted p-value>0.05)
+
+versusDMSO <- paste0(conditions[-1], "-", conditions[1])
+trueDiff <- TukeyHSD(aov(
+ trueEffect ~ Compound,
+ data = dat
+))$Compound
+trueDiff[versusDMSO, ]
+#> diff lwr upr p adj
+#> Compound01-DMSO -0.15660919 -1.881368 1.568150 1.0000000
+#> Compound02-DMSO 0.16041021 -1.564349 1.885169 1.0000000
+#> Compound03-DMSO -0.26495737 -1.989716 1.459802 0.9999957
+#> Compound04-DMSO 0.12799598 -1.596763 1.852755 1.0000000
+#> Compound05-DMSO 0.15392185 -1.570837 1.878681 1.0000000
+#> Compound06-DMSO -0.29447551 -2.019235 1.430284 0.9999872
+#> Compound07-DMSO 0.07293908 -1.651820 1.797698 1.0000000
+#> Compound08-DMSO 0.46903318 -1.255726 2.193792 0.9988016
+#> Compound09-DMSO 0.49798875 -1.226770 2.222748 0.9979416
+#> Compound10-DMSO -0.09048129 -1.815240 1.634278 1.0000000
+#> Compound11-DMSO -0.47509699 -2.199856 1.249662 0.9986527
+However, calculating the differences with measurements reveal that
+Compound 6 would have a significant difference in viability from that of
+DMSO (adjusted p<0.01).
+
+measureDiff <- TukeyHSD(aov(measurement ~ Compound,
+ data = dat
+))$Compound
+measureDiff[versusDMSO, ]
+#> diff lwr upr p adj
+#> Compound01-DMSO -0.6146694 -2.514910 1.2855708 0.994474856
+#> Compound02-DMSO -0.7383338 -2.638574 1.1619064 0.976093386
+#> Compound03-DMSO -1.5752825 -3.475523 0.3249577 0.204689940
+#> Compound04-DMSO -1.5418117 -3.442052 0.3584285 0.231362089
+#> Compound05-DMSO -1.7724524 -3.672693 0.1277878 0.091021614
+#> Compound06-DMSO -2.2208497 -4.121090 -0.3206095 0.008978303
+#> Compound07-DMSO -1.5968686 -3.497109 0.3033716 0.188684312
+#> Compound08-DMSO -0.8412919 -2.741532 1.0589483 0.939725524
+#> Compound09-DMSO -0.4007553 -2.300996 1.4994849 0.999893315
+#> Compound10-DMSO -0.5485415 -2.448782 1.3516987 0.997945430
+#> Compound11-DMSO -0.4750970 -2.375337 1.4251432 0.999450309
+We can also detect the difference visually with a Box-Whisker
+plot.
+
+ggplot(
+ dat,
+ aes(x = Compound, y = measurement)
+) +
+ geom_boxplot() +
+ ylab("Measurement [w/o randomization]") +
+ theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 1))
+
+Given that our simulation study assumed that no single compound
+affects cell viability significantly differently from DMSO controls. So
+the addition of plate effect causes one false discovery in this
+simulation. It can be expected that the false-discovery rate may vary
+depending on the relative strength and variability of the plate effect
+with regard to the true effects. What matters most is the observation
+that in the presence of plate effect, a lack of randomization, i.e. a
+correlation of treatment with plate positions, may cause wrong
+inferences.
+
+
+Randomization prevents plate effect from interfering with
+inferences
+
+Now we use the all but one assumptions made above, with the only
+change that we shall randomize the layout of the samples. The
+randomization will break the correlation between treatments and plate
+effects.
+We use the builting function mk_plate_scoring_functions
+to define the scoring functions for the plate layout. We then use the
+optimize_design function to randomize the layout of the
+samples.
+
+set.seed(2307111)
+
+bc_rnd <- optimize_design(
+ bc,
+ scoring = mk_plate_scoring_functions(bc,
+ row = "row", column = "col",
+ group = "Compound"
+ )
+)
+We add plate effect to the randomized data and calculate the
+measurement.
+
+
+
+
+
+row
+ col
+ SampleIndex
+ Compound
+ trueEffect
+ plateEffect
+ measurement
+
+
+
+1
+1
+1
+DMSO
+9.530078
+3.259601
+12.78968
+
+
+1
+2
+24
+Compound11
+7.810705
+2.850439
+10.66114
+
+
+1
+3
+57
+Compound08
+8.419993
+2.474874
+10.89487
+
+
+1
+4
+4
+Compound03
+9.640621
+2.150581
+11.79120
+
+
+1
+5
+91
+Compound06
+10.952227
+1.903943
+12.85617
+
+
+1
+6
+94
+Compound09
+11.928334
+1.767767
+13.69610
+
+
+
+
+
+cowplot::plot_grid(
+ plotlist = list(
+ plot_plate(dat_rnd,
+ plate = plate,
+ row = row, column = col, .color = Compound,
+ title = "Layout by treatment"
+ ),
+ plot_plate(dat_rnd,
+ plate = plate, row = row, column = col, .color = trueEffect,
+ title = "True effect"
+ ),
+ plot_plate(dat_rnd,
+ plate = plate, row = row, column = col, .color = plateEffect,
+ title = "Plate effect"
+ ),
+ plot_plate(dat_rnd,
+ plate = plate, row = row, column = col, .color = measurement,
+ title = "Measurement"
+ )
+ ), ncol = 2, nrow = 2
+)
+
+When we apply the F-test, we detect no significant differences
+between any compound and DMSO.
+
+randMeasureDiff <- TukeyHSD(aov(measurement ~ Compound,
+ data = dat_rnd
+))$Compound
+randMeasureDiff[versusDMSO, ]
+#> diff lwr upr p adj
+#> Compound01-DMSO -0.05930995 -2.268768 2.150148 1.0000000
+#> Compound02-DMSO 0.10429864 -2.105160 2.313757 1.0000000
+#> Compound03-DMSO -0.41812087 -2.627579 1.791337 0.9999637
+#> Compound04-DMSO 0.08581910 -2.123639 2.295277 1.0000000
+#> Compound05-DMSO 0.18471545 -2.024743 2.394174 1.0000000
+#> Compound06-DMSO -0.33999104 -2.549449 1.869467 0.9999956
+#> Compound07-DMSO 0.08156504 -2.127893 2.291023 1.0000000
+#> Compound08-DMSO 0.35967314 -1.849785 2.569132 0.9999922
+#> Compound09-DMSO 0.30525718 -1.904201 2.514716 0.9999986
+#> Compound10-DMSO -0.05702472 -2.266483 2.152434 1.0000000
+#> Compound11-DMSO -0.52416017 -2.733619 1.685298 0.9996658
+We can also use the boxplot as a visual help to inspect the
+difference between the treatments, to confirm that randomization
+prevents plate effect from affecting the statistical inference.
+
+ggplot(
+ dat_rnd,
+ aes(x = Compound, y = measurement)
+) +
+ geom_boxplot() +
+ ylab("Measurement [with randomization]") +
+ theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 1))
+
+
+
+Discussions and conclusions
+
+The simple case study discussed in this vignette is an application of
+generative models, which means that assuming that we know the mechanism
+by which the data is generated, we can simulate the data generation
+process and use it for various purposes. In our cases, we simulated a
+linear additive model of true effects of compounds and control on cell
+viability and the plate effect induced by positions in a microtitre
+plate. Using the model, we demonstrate that (1) plate effect can impact
+statistical inference by introducing false positive (and in other case,
+false negative) findings, and (2) a full randomization can guard
+statistical inference by reducing the bias of the plate effect.
+While the case study is on the margin of being overly simple, we hope
+that it demonstrates the advantage of appropriate experiment design
+using tools like designit, as well as the necessity of
+statistical techniques such as randomization and blocking in drug
+discovery and development.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/articles/false_positives_files/figure-html/boxplot-1.png b/dev/articles/false_positives_files/figure-html/boxplot-1.png
new file mode 100644
index 0000000000000000000000000000000000000000..a3c9937d7ce8853e2ebd4d6e314600f59d1908a1
GIT binary patch
literal 42337
zcmce;30RJ6+cta^88XBo8B(E8QKlrCu*h9BY0^xJ5~XM!Zu2q~Hx){S=A=mTXhmtJ
zK{S&}^E^-geqOEhJnQ|w_xs*&`~Ua<+P1aTa$ol~oY#3C$G-2!ew;T|l@%6nuH>XB
zY60WW@9GpaBa{3$XBPft^vfb&{F-$_N#S?=#?Q$96nFfZYklaLEk&)qLH=X&Z0s?m
zC;^J``(6zPzxEnOm!$*Wr+S9WZ21M*4{r1km^EiXWBhWBrH!L`_r00>xbmOz%1P_S
z+$#T~tGbx$r)W1fTmAx(_g9YU$uC^I|E<8s<7z>IHcf(arR?0@RkH;yWp)-6tsire
z<&)(LXfG;g?l`%>xAyVV^NBU*66pI1`+m87A^jIqT$Ra2{$jhtKA-%aGiRwW`7JLW
zNX;OBu(NY4C%+dh+L%dxGnw*iv=cl;bv2f98Wq)aN`>Xl4ZrxJ<
z{r9Va!s6m8LPA12H}rCS{`{Gfi_1r_v8gHI?c1`Z-zP6ks3$#owEWt&YYdi+jg5wu
zmYMC^Db+}=&DXA9S7g13jnyr=QA`DXEFJHd=(=O#=YngCL!|>hR7h6Amw~jG#
z`#J7*K1qMK>*(8>^Wz6=k}nD|8XFt)ii+Z;3LBpmx6bwW|6y?c8^-tl>M7pY&1SlP
zORx$^AM4!1P(N|v`KzR+mA*^Jd}2y%%uLM8zPl>!^yjsV5~Hb6?Tek&-;swC-|K4VdOcB<|_X=>UM
z>|T$J-<2KD)e#34ctpFjL3CexqNwt+qlEEt5=6NuBbPfKEZ-dAxBmnd9s61p>KO6xY!6w;mHnj
zR^?mshK7^OsX6g`Fs>A3`}RzQa$i$n>GcJx`pdPAgg0zZIDh`UMR|nRxr_fIL(;c71xswz_*V+Y&6yHI78ru|bQv^t7IV
zp*r`j&vu-goc(#C(Z7oJbd?#{*BqdOIddiRwz1|9FKt7$Lx=w8%$RbM;$4{1K%eBW
zJlo{`J9p-He0{xt$@;^p4i1Sm=jl(@hpUCkJ4Yra>J-aOrIKg$SiPrm^=!)GPVA{-
zi?P8D1k{IX*RCx~Ja6(LC#P!lJfw}~oYow%W`*kK&%bt?8mn_oy^E8lO-+t@$?~jN
z5q`cQ+cEX5(Hzdv??KF%m@Nks6sjI&yI;L}b$P=df6PH9_=f+DF%se79nX@BTD*9%
zO-jwqkk{Idj!DbJ^uw;rE>S^Zn*-QK25h9p#Ugmef^FJ;Zezp;m4v>E
zdquX4T`Q)PL^f|O4`Mth3t}u8`1Lufa~Gw$uHf(`+>Ysn^yUQ;(`TRu|UW8Wt
zoS2{ww|PV*CAE)+9PSy7C{Cz#ifk%+s@G!s?%lh*+}s|EF)dMXamnltA3nGRRA`I%
z*XpUN2EKW-&3DPjjgKD>HGC8wYn-j7rk36EePm>WXI!7-*7N6U3j^e)5X{`h6TQU@
zA|5_mdi&@|=&xU=8{Uqjrb;L2i;=G*KR5JuS6AyRFPMtHHFmA;vs{>Fr1$w{t8hw;
znawUI7dwbhG8YB_51)sLfzy)`PCZ+ZrliADW-H#xv#LB+2
z!uGRm^VA~YjT^U2?s6V!^tm`OP@$b1g=rRP*HP)?*Y-2_fni9T)(5#Q`eHoSnleN>
zabBCje0R<9{ZA))XAI^iwXIpo!(-IbOO|6@y6fh{KQlxo$JDj74zf)~^LO!Zc7I^+
zsq>CNG8xPl!lcS^3)eqVE55XzGh(;ml0OvAQ<+P0JoytYIAWkxRGC6)y?LTOE}b9m
zuUomt?&b66&)=TR{f$3_o!Q!EA?~mCKs+7^WqdZzyNG=s@`GkjuMXc{wi#g@D}9MX
z>l|y$`p&c=!-lrLfERfmKPJ}y5XqRD=(?D{Ibm2jK(f}Q?}+PAjlFlV?az;kO;U04
zD1($Qy;3O`TArtLCg=A$Qp3tG4&E(GzIdkNTWs^2{HGbal)7ruTpt3z&3v;`lNqe!
zuAK0rp}c3`$`reflG-$9*WJz9My`zy99t`*yB^>S9$DW%C7#Dy%i=N`JfU+gK~Gp%
zIJqZ#mFuI0I7-dikxVig-V~fB$uLmgpY)&N8G(EBpD>GNr?Z%c_%X<1ETWOi!Nt_4T#M
ztvh$h?n>5%;pRlwta;b@2XG`)l^OYL{gEdQ7yC}RTlfD`OIb&C-^b((4hgXs?*7iX
zV1bvQ>8Vq;XQU{q-d3MuC5b}+8(eC=coR8djxnB(-?jgKkZpqZ*>7)x3va&MKOOd{
zCcBh@uO^-h){~uQBOY%*Rq6L}w;%fg`fL9a626=hzy_?7q6rjg=B+sWV#uMCL_iLQ
z!^G$S)~v*K1UdY8Z*3I$B;(Y7FYEgjlcTJli)~*X$8XM@ADuoLQBqPO$at{+h`PEu
ztMDc{T7UCV&X{l4$_utFc=RkaRUd$uRb;Rp(}t2C$T|>1j`;^h=L=we`uMTAna4c8
zUFZLNa~3ZENc}+$N8-7fWV5F3Z)6tYrQD?P%|E{xQax)fK8eXszSyRrp%D}qC~&c(
zvK-N6d^w%d%PY@Br-gK7Oc^4YAvyY(9%B;|7q0-88+bRHf8)A!`;Q$vh6Szp%D%gP
zZ{%9~O6-q>IEFJX&1n4+<@@H%o0ngu$@MAi!hUu?U*E{Vj;c1qu51%zEY%weS1(<*
z%xv%ieTe?z(s|vuq}jtdv%)9lZC|1yw4?pMmC_$tYaY&6Q1Zq!|4vVpRmPDK4)U+V
z9Qeki8ME)4$MAjgrITakx)7dvOw^Sop{3@7P4qo=|IxzXP@j1zdzYMCU3(!tM)l-s
zkG7iJ>A~|C)K1?e-_|rr(xU8;xw$zjXsj#4XoIrf7H2~8sP2!Lw-U~SmYWTd%Q`p4
zTxvF&pA4D!EJYPRqv^7|j0tW@Pv`qG4|D4dp|?%?3ym9U?_jm*Q|
zd~9MHH-2i1&Ul4W2=CgZbN~K*tV0b|)qA_`+QW{;nh9f|^5&2;QitBeP@%!WEtn1E
zGq|J!`};2>>A#xiUzcPXdeDtDCU(?3OMM*g%(B17QZnWVtl;m~a?`h)(Ou^U@aetv1--Ya*ayHdJjj|9jXfCKVu
z($>)l>uJb2(^T-774&TXjfhjl{}d`?C%nA964Xw37
zJN&RVZOq(b*S;ki**Vf?uCSy)%MlT8uNWbvGxMoCzER3
z#?B6O*65Ke|GFBp`Qg2L#&*2)mjxN`v@H9t@fd!(xn#qgQeR2SuYn6z?bNp!DJ#=B
z;=8l()<*icLvMf@E_AjQ1B!lXD)ja8^K;28MT9V<8frv?b}JzW1P$DP>xr5)d#Ulk;}IoAIreMgz)^*axd%QKnvJRHO0pX;DO_&y;2LRpAZ&9k<%
zi{F}#yn6O9y$Zu}Q%C1_Bad!QsB(R0(kyN!#26du(pFV%TwMi%`TpjTC)-SK_2_u=
z$)w(#zJ!pYCUeP>C3j1GL`x9~BP(l>KhysH@S7Xw+Y7HYT77$SpwL$`VGkX(0<-3BTe2Cz?<
zD2JAA%)~c(%=+kmev}=_uKzge2AA*OA;-*#?UhkR`oAKQ+xh7%OT9E3U0kmuym_-G
z@`G3Yf!}{07N8TUz$YCoCZwmGn2O(GPn{``(9A0?&iFAvUtx={5C;OZk$tUWs9pQl
z;~%DjccMiIk1^tyzl`(B#srfb4?@o`*0~O?NKnv?w|rbuS~^goBp!}j$t5=)b1tQ0
zXY31dLixChy+0Xvgxp?75&vtx$UZoAPOp?uxK&*`FvLKvelr5F+~lx{Yj5rbR?yF%
zCvi9f)C$ij$=Qu?wBz(A0fP89>&2J7JQlmdqHH6Cj6d7ysiwNw)YQ}*Y)kfOUw=!f
ztZyvBx~zC+h!KYqW*Z%4ATs&r$0Nr(4U}tZ>g&IZKU*atE?zu+21M8TIo()wch4i=
zi5g^g*dbURVgV0an3@mt$ZpMoS5v%I77bhqgp=Ydw!JEDepd)BSXT5b=e=U)#RNZEJQ)+`SV4PDiMq`k61
z?Zv^Cw4vHk0C@@HOsb*h{yK#xAdTdQv21BQUWbXaRjsrMFF$|uh5#4!lB8m4sH@H>
z&3Wic?t=|NPg73x*T1{+&)e14_d|Lh<7}GmZO~m^sv;MKyEnS*hHgt9{Z=
zO~>upEfYHru>e4{S{fGK*uU=Jolh~xvo0W>c|CmiE4l4e%FkZP-|zSTfVQq|J(
z_7*pKRbQX4V(Ye=KL5)TEyABfwO?r@oU75$*SBokLI1NxZpm%qi3a|uPhjW43s>(x
z4f1m9?%g<>WQ#RBc9gvsxO~~7Gbiq;a%j%jhlSvGG&6>60q>(S$=r{RxoN*G3}EU1Px!+sz|W@JWC>
z*QS~m_SNTaq!0g?2f6N8lwN|Clao`X(L|C~lrCSC_Jb+a`B*^?8N-SnCis^vYZ`KN
z4gSI3`ThI%b|1y@;q>NR8+Pxm?408!yncN)e%+i*;
zcWoQ)K}ir56#Sj_(AU@b1CQd%mJ=sV+_4}2CQj4Uo+cAz<%J5&smLF0-|MO|IBtu0
zcsAU~T7xL$=f|7Up#LSI&Mgu|d+b6H2zWrDBZ!)F|B$?!TRMmNxB?)6!dT;zGp~9a
zlmmE=S^*ahcGX1_u(fro8lj+=hpt=W>U#RQzCRhiUDsi!9eKIlCMiPGP*l^!;Ej=6
zrdv2s1GtBj937=NhBMq!jVyY*_Nw%3*N_@iY8-dXDC>70!>GxO^2HbxG{-*3Yby(X
zVUS|i@u!^pm984hVp8)$Avlo6{JwxtCsfwn8
zfAbHm)EiVk&)KdYb$pFW)^pr+ac@OjJhKIeZ!}#tb
z{+4>K$>AKiR`8QJ7k|KRlktj-NlhfuUa8p&dBXs^@|)R=qrF76TM>YL1Sfv*WDpUJ
z5WldbO`9`u6r@?-xAt9ztARpqy;_(C+Aq59+4bDwV(oM1Ua2&1+PJYhSDIT%lKG@9
zi%YJoIlx_v(OMeHMJU_6W9m%L)q6tl6m1B6X{{H9GrKh|TzHG1fETE)tE<>Uh|{JM
zPBa5haUVVURNAq`VUvi6Vw7&2x{l7HZ>4P4F?yO~6<ly$t!i@PSsm;lfkAn=D^xuA8+?u*M?|%+J0u{HXxx241^$K|MmH_
zq&_^ibe?sjR@7VDW@U!D>Y{`d@qc0-;b3xaVRpc3ICLu~fmwj|CWA|RGW>N-Qar042!DxsPxIvGXuzfxOxg`R|v9bE7o#Uuic^W
z@Nli)+1bB9`S0+kkE=O
z;qF##OM0>l&jT~&+hWW#bQ>{cs3VreW$11tr!+iv?6C)*OlxdXkgfK7IhWxWm8O?~
zM*H^fciqNA_2EPccUPJ&tp|GwR|$|MbDW~&h@d~4&ml9^*4x{A)UpT=sSVYq{#<>k
zz+_bT)7-gpN5|@>#2BMBU2dW6?Uu2jW~PDT&3RY=lw*AfzDD8g7JMhTp$9K5*0gDrC&mL9~iFw{whN1?5(gx16z
z-DQ_j=s_NMm5`v|X{-qn_X8&THxRL^b2*i2j9a7cV|ZPuCuLQWT;6_!9>#hJtTiN%
zf?7fPbHj$equ!Q@IW%<2)DLhQ4PSrwkuhe+YFU@BUq)PMU=Qde!~g`)tX)X7CCXaa
z%q#>XvBN{HW7p=*
zM=A`Rqxc-^*@8nu^BZjpy2p#2?vZfnKV2TBr=9P;IaY)&AA+~<=GcwHfp2-Fzl5sr
zW2qYg)ET+-TmxVc^2rZov3{$uEtX*gVV>zNo<^B{^Q;XG{bwy$dD6udjKUmx1sH6)
zaniJ`Zn*ta(L=!nt9NUF!2@6TeY;QJ+?EVV+%Q!qI5?OUl2bdL)>RMG)nx+=ZEr1
zdjbB}`h!L@2<
z!EzP9U212(yiEG}##hqn3v^=<10H%rwi6*s;!Ft2EboVtu{M$$JodbG8L=TMFT~#p
zBt;rp0Gvp8ui9>Q=eOT}EBF`9=$x^MA)eZOS=n#balhJBgwnHLW4^QUL!tMgEUEQ7
z4Hcl?C@_O)-gf`)9MFf*=jb~=ra|s
zD&LsiMy2cFWVC4=~q(v_p8d1jGxyLI|%fLeEUSUL@hk16B)U>Ef
zxo6F`!qp>gqO4)kZkN(%z030O4658ShhrG&Jq{XRLBU#v?wgZ0TdJ_$!se3c>e^Q*
z$qE8rILc(J7je{Q`fSzsLROOrjG?qRr2B^1AX`}P&kVP*K~It7nkUG7sUHCvg?
z6QgJE)jBq>ZYXP(taWI3-m8aLU51+!Vx-u2HT7lCvm_r8lxMT<$vb)p=L{NqH|xIL
zl6{*;eCNNUvRxcb4cT|)EXTJgxEI@^PUlb>a>OiWFZC_%jCP#dwMl}l_K+y0YJwRj
zy6hf`Pam?g4edrQUuHMqUAfZ2j(6t0&o30JR+A~iH
zV8toM1~@=q(S`P}YVP9={2dN*VoevXIvdI3`GMA4DEq~2P;>zC7uW|n%Nd#b?zwCU%{s-Lh7g;N}T+-
z2rNpOlo5f?2EO0mZf4+4cv*TX6bmR8zgrhtVgs#`NL2l$G
z7}Copd~Byb-dh7k6$`BPCE7@ZhTfZgJjgPI?x1++km|mDm&HS9L|DJJe9YxDr3tYY
z!6?PVp>g~}F}YZ#_QuR|)c{%7b`SEuRJjL~^l6yRf6Q)XB6OTplIO~1^ScD191MX4
zjvei!T14%rTN+UlKR3vsIC;OZ4~u`}o;^_P;NXD7q$7I@4199{xp;XNa(qWE`N%L5Y~1q0{yc@
zhTZOQX=UDjUb%>v*la=WpDOnOGLEY-+@5Cz5%=xIHZ7tPw42i(CW>G#oPj=`Jq#om
zGEP6}IMEDHBgx(+ES&rAdW6cQb6;bg5s=4#Jh#J{7Tzpkyf^Ei5MoI3_bqPGevq_p
z;BA2Le7mdz{hd3d80z4izzU{7z4);RgR*a!F(bj`(h6(4M
zsvkXC=ud8>{LsOJ*OTqLNL1inwhU93>4SNHcclcV!joC^mj{v>2*#`wk9TH}5U#NG
z2zcPlXHd;U8a!ocN=G2rNBkXr%#}D-twZ8C&iU%=R~;!{k95Qlv&W2DeZ%xN6z?P#;yRJc35Ijz-6v4l)WJykC4#i$Ed
zlQ2uf6;Lv}rh*kHZfX~t;z$HJ#cbCU<5|4;dk^n_>hG}hYD0Dq+OVN*u*#}pfn;@H
zV&X0i4vz4kpx;|sTAu8_7y;$Q+Uc2BK1Vx~p?T8_Io?x4zuIdGj;>)NsyiKCXH5j)MNf-80Z%s&WS9
zfaGR>=Utq{BIr4jgF|@d&MzBmvB}zVQ#G`R_8oM*I5uE;c{4w?Hh>
zvaD@}b}8DTFI5aS3p7QuLDkg~r$5O9%Jt>yUzyFN1(u7c+!!mx=6-5aZwCstqOFPma&4T|h)%f<1W|uEXDN#Mv||M(HP&wC@TD8(#4PK9N<^s=Kr)RA`dI0$h4;
z3JOa}9YbWqx+?4bo^^Mv6m(snfI|p~nz*>cnJ=3#?G;H%+T$S%4Riy;0<{^h05w^g
zpMRu}_vR}G%v-QP6CvVA@u~GUvX0-ma^=cIdUwUoQp?nDQ`{MzDzYO%$L?FgoHd@a
zT>D0a>BTXrUawnn(I}s48U}7z1Q%AfQ~2*XI?Km`e*^Dun{eI&9j|kaH^k^Dt!O3p
z;=HzCn>B0JEDP%4w;LZjheCs#d-_&?I9N>O#H(NcNgHjAOom0K9EA~R1O=x8R1(zz
z709a>Y(ZJ}rOgQBu~sYhF}>hom^@luc*<^*2>2%AmH{5ay<~}r$&-kYK5;iI$JnW2
zA_+#};UBHYwjSm>
zlpSw$qQ2M>f9Yf2zI|)0N{GfwHMghv2J;34As6C#;9b4?>>bhc2hNig7g!ph%BvfC
z-oD)-)BkC|nL_U$aF~^aA4QaZ2A%ls@~uB^eL+FNi8OrsPI9MBxS8vzcvkY9FFOEp*hh>d;^PbICfH!Q~3`xrVgNHL9+k&pj*Rdn)0
z71B|@oA(HKwb_TaGdaGxJH$?%>2(9O=^3jq{cu8ib?>O6wRMad6EXBJ)2nLFfO^ej
zV7&T=i631DL{LS6ARrrHkD|Bi<7N~ARn^r?G^3ll>Ljf+69Tl|i&xG$Q2e0l9klVy
zhZ^LN(`AxByp-hl+Mloh*#6udQMW80Zxf8al5P*bg8MLU{zz|%dFlF#08Cg`iAF!^
zT;`C2xGGua5-U5=f0R)I(_dNO-Wd$_Xv4Jr$+0da*3Pq)L2!Lmu!4}zO$<_^@9DoD
zkB7BTMY`vzLf0W`^3U(qr-AcGbO#C=sJ1Oef}`}I{ju>5hIvn4?c-6We}ng>+SySp2yfUBs)Mdy2Tjb=)4XSp7a37FD&EXX@jvwB3;&f4Z%7Mc1o;%CQ#|UEG-7?CC}}_sS%+@pyBYUs<6_|G)(MeM0F&FZT@06zBkIM0B4*5*
zza`tz(ed9pj2mY^Vy}c_?@2}ptN=TJh=~;9S>PwtcUdpym??5>wnJ^IBdH?06yZHb
zLeKUt#yzV({fl5q**2Yq18Aj`O;EC3M+8(WUzr7zejMs^zYq>&pV}go$v)n&W({p)
zSA|2w8h{U$i(p9679hSBKp=ds{c01vJ{cNt>TvVAj0m??L|aW4*r)>WOn76}%*|Uk
zJdZryb^u1Yq~`-k*2}0!SPomJCcCBx#(*kwrsu~88bF8b#Pd@1U4F`;Zlk3EM4vHo
z8wsvijIoH|hR@_*P=-mJhr=8bmndD;i*w}gSbHwEo*Z1yMs+7aMUqD1-p&QHWh$l6;w_$
z>!!{4Un7u`ArO12G!aMy%N=Z(7sfgg#!eP>cUuP(T!~O)s+rtbC0Vo~bfIo;#ZVZB
zYFxnh&kar}Y7kZgGZaxc@T5#d%&Uu}{_47|LV@2k2*@7nYt&MCn2U3Z#U0FHfSIsM
zkeg}683b(ZzxxZUXD%y87~957`2|BT0RZaa|GVe?%R1&_H}h+U9jUjnAdwcyaI?)L
zN-MCHcX2E@5zO53tTG5o0s6e{p>h-9jkY*JpxK+Z6wlJ7r!fEzyzby>fUsLPZ@%6(Ymqg?^yUYSC;>q~G)!|6UbE)ihCcRm)cg!&#@b^^@XPZO
zy<3&UMlA~4B$^3=myAVu_eHu~JVft~>ZGGwGX3|OiUXpFfW-~1#}o5094mx;7v<3t
zL=3k)#b8rbQaY-rn8)Ec&{C?q6;hl&w%%G8*KesJd^SnhjZ7PGFEMePpm{IcS+5Ym_Qz}3dkN-yN(f{4v8_@s%
zRMPT`wQ0S@`m5(vfht4(+V9UcI_-mBO)j$D6ee^ne|PtgQ$QCNHXwH9>5ZT0#*hw8g6xNRO%t8
zZUhX>YoTAv79Cw?@of(et$q901L*1#^%YhU>9dtPFn~FUiQEuI1qif$y89vttkBuO
zMW4i{0bvrU>TSoj;K0C4Ea23inr!}Cj$h_;8$OPCXioBgv_lLrM8Iql+l%XItOA1n
zl?tPgC6o0ot5~I?obSqNz!p$u=AXx&k&gcJdV|
zX(RYR+a=Wzm8PEpT&gmTu8UtHMw40qA>q*W<4zUxf$gXh+dR@5K!`>4tZ-CFAo{;W
zhzTWbE6Mp3+eu6)Xg`kxTfFtosiC_HWJ@QUu(Lf?O{m$Hq6Z+lzP^6t`RiO%+swzw
zuLA`@BU-{k{neq_<@!E5}!jtK$=y7KkpN8YMg2QG9W2s$Uino`c7?831taZ
zM#w*{O}vhC>JfJC@7UCh^wLNfF@F^IegJh$ei%eFJUylvfS-5Y`y0>
zJwYut>Tz~+u%QS6dM`8h)npXh+6!_qKK_m7@UqM+Y7VzG=FOp6xXqceY%C
zltTn=++blX?J}ke5mDuS0IL@XO|1
zx-HiGU#r$h(T78t?mIeD>=Hn{Kdm$J=X&M)htL+7;>Sggpr
z$7H!EXCkj4xw8K5+nJF`ys4{pT81I+oawHAhao2egBsz%w7#M}e_;tIBShujY?Q9g
zTDoM(3)Hwt+61g34$b;);OW+Tr3S8{aiSPRgv)*SN@sGL7^>Y~V?zns7LwN>z&tj%qF!;syoFlmGy84H
z5(&IYBCJBK{X!aB@|OdrT)loB_2zRVGh)R02P%xJ5bUVoH38i9n6^lYD#Qm)>NH@{
z`fm>-@3?Z$rn=Wb$b15GAG8@%7^F$-SlB~Tj<2xfkySvv&nRJ-V>$9{E=_f`CB>mM
zr-71Ux54d1g~T~1hwzs#4+D%q(Rlp$arQ95?ba~^UEe~3f{3}6AlewumW(6BzDW<}
z>l4>7l41^Gs7;`bO&c+wK4P(bX-61k9srn;9q-;BP@F=SnQbxo5kCU2L*C|EvF&%d
z5)~G
zDzRZgYcB^4e(OHv2COV9BU7Uj3TvI>kt5}QEME8W4_YOhV+}O$>M+V9q``;y!vOt{
zLg^y*LZDMpL-i7jGRiPSrK0};T^Ew4dKql5K294ffzs}QTPZ!2b*-!*SRE?i$KI5@
zl$vv?MOoByU*hRi9v(Z5{b+U7tI
z@ESJn-XqW9ZA2Hem|duHp|$m6haLPvN9}Uh#xECl%^-3f#at~d%Rb(j(v{Z%CJOo2
zSXY1nv37g5nbj&vH3_9_k{Yq7bWtfM?1mWtuPusnw6W&ST1#(eJSLWd=sEDRO{qbT
z)fJ?C{_=9mYFT~NSI#i$)VdsuGY9i%eYhIO?#r78{Rea
znA)2)XWckrSG%X~G07?Q%gM}7HH=2pq+N_-rvP2@v%W)gd8kT;(c0*Ic$30AbH$Jt
zEu!pVXQNxI>*vfgCIi7-w}?q6Cl7gwP8k370sDf=#djeTI!f~~O0Y=SaBF#5v8dqb|b;LaGFRTS>6e#XKodKX^p!d{kjB
zeqePoF){I8w4@GbXTfQ%dLpJf{clh_=)|Jz&hqp_DFwWGx(
z4!jYB?Ex&h{N=Huuw<1BV+~<$l|~FM{dE+{>Jb|;x0O3i%Q~A(Mwb3UV$_>pk;5nr
z-$pjlM5L8L%d%AYvh<;u^rQ^DdM}Z4$36v=YLO$v8N7u^dekCA_GL^NKF?*Fb%T(H
zjo{WIt=6I!=dpm=W<>~U>_dDey?aV5kgt`4LoOtY*YmQ?%>>RM%|K|l?vziJCRe94
z%^WEry+47g0FU|Oi#%Lz2{?}YXgqFIVB&I706{JIN+c2U
zK%{<=Mss*^9{Txh^2yiyK8l^MDfu=gtWF)NXGZmw)mR~OyVxvU1IQ9Puv}{H}4AP9Z
zAmIQ%c4AUoorC!UO^3nCX;6c!rI4G$qw%k>G3S6cKbv9lh46Cmxs{As
zQnA*afR4~pj?_7H*wG*NI!NO+de@lZ1A&i^rn~B_XXblSGvvco?!It3{B}P_9^3+?
zFOX4!%98@AwbfU>MBOtp0h7);ex4#Fi|*5fyt|A$9YR7wXPd-Qs(Ge92Fo6H5KXLU
z7)=10y1Kfm@z7wIw>UOYtFAwTQ%(;#w*r^XlWyLA{Lh(mmk04k95`sBN*X;V<1}Va
zA#mo!mXZU&K~f!WpjxhkMMW?iZ^Qx~Mghg?D6KCqcKITtGvDu|4?#E8;dS-7DPMT2
z9!MF!J*_5?M=y`eQ7RQhDFRzrWhkn-;_irZ83AxyEC9`V9&kbOKL
z+9sV|a|L+cXJ?DRn~X3KDzktb`_5#<{~M6&Sr5_e!AU0cQY}?0zvK*%KAe6+xDRS9uq@oQ7|c_Qaq8}Ya`9uq5^%+_*0L#lPoPQ1thr;RW*Qn%Mshi+Cry^iTzwO
zy0R_El*QwtLir5LY5AQ(a0Cisc|0EQwEc?mLAXmqL`C!8R8*zArVzC|-;j$kjtV63
zVv!+O(?!73uLcL>h#RnknorF+OFm*+{DHk#&~kM7O%yu$^>J-4LDJ4B=7ZHJ4cTx;
z`T}Z>H*;1!*d9l)vI$Gi(0{)J%71C
z!XD#_?HtUQrT3^*IGl;90hAlrn>>wqk=k@#lJ=fm5BpITZ!y*$l1}`;qpP;^FhXhJdC;;b
zx9Nme^am0gdQdhP`H3!}7x=!bjdKU=Gsd1u6=8#y(MI~LyVl{XaQOj+D6;xT+oJU*
zdhrq|%5QQ03~NQjTbLCgn7)L2p~L&lKdyPnI|wR=I;tq@U<5?MzxeCZUedh+o?u^D
zwT8gyOB-o~Rd000WH$wylt%nogbPL*(b0>!
z9f=aXQFh_odl25RLBM|0b2Y%z%6;in7O1|7D-*SRxq=?d`-@d)6ON_PSEM2edL!kd
zQ9I{_JFM_Oq(f}0W6O(L@xRj{S0*e(k~s*C|2%LU4zni>=!EzWg8~5w?(OxA;mV*1GRZicp@eq#{NF)Sm%|tiQ>nO45n{1n5
z4KE7NYe7bHQ&a0u#*}Mth2;FK*6Bfsar>(e;Mqs4V-T^pQKU5dgzd!J!TgGHW6&3m
zxcisgRbd*9Q@rnbg+ER5%DL&ocE2Th;ZuKqzB`5VJTT_ep?1#SZ#eCsd|6C(vg0TX*Iwt|zF6+I3jI#0)@y@LY(0es518VX{$WGkXN)X`SE3HPotGdnu?DyfJ|?Xj
z*zqDG|CE{86BBmYgDaql*D9Vd{zi&V&dnuB!OfL8DtpiZXZ)|+AWspeq!jVoNWtvw
z<|FR-ITBKe-oN7bB4)z>m^{c`SYHNS5EA;p>n!d6V3o)zkqfAON};Y0n>@V`1vvyL
zBiDiQro?SWpD6KdJqLICKav*lkv0)oOs=CBU9mfJh-3YqpChoKFmhjzbhhrc`T6nY
zf|1AUPr>ydBP+3BP*SQ{LNg*t6bSLVwlq|n920oX3(
z7wCKmgb&TmZIJxSfH%uC+^6Kd=;_1^qE@pF!dISd=-eT3%xs$`Ix(;{XUU5|&?85h>?qH__MHnAJ;F1EC|;rNG
zx}QFGZj0DV(-kz*JO#J~!iH3uFlx$1#aydsgGFboT@0&QC$pFG^Y?!R{;$ZIpLH+4
zkq$dE^}&dx`BX9!j_T-ajII!604*=fYhE8v2t@Wqnr^_GpNYXI(kMZ*0`<_KIsz?E
zp;9#p8C8&h0Eyy}B0*8iCCA)W1z~+YZh)~cliVtP>hB;QesG&K-R7ktv&glt#Cz|ooJv#w`II~
z+p6owFHSF99l2@ofT?V}4lyqj1@&nQAk=%+CWyv|1u2|666T;&07^QvSBD*z0I4NzQ
zb{vT;MSRR?w>JpCNIHBVYUZ^KL;#;AgWYhW%^Go)*MUhW8bn
z_=A(1hsS3u6|)FvgY*qpa>a5M2FvEnViseeC%rHJivD!ObT%$!R&TPLM)E5C7D&bfp=KilGF
zxJrfp9BZ_6aGyQTwWq7BPTbD5Wkue;i6N!jsVwUY@e`@nN6~wEs%q-^md`v>;PKz9
zd8(|gz;15tfdHwg$KDMehtSjcNt1xsiYypy5Ha!=tZ)T>|1
z6oh72nOa2>uusid9na5JR9q~XV|V06-!^FQ;X($2L%cCdE0|Aiw=Bv{=jQ|t$?Itf-ZmSrDdIR4FEPs
zM1O$RSE67wCo^MLv!5-L#*@plj}5plKMfENB<}W!XR1hMNT0F6EakeBkxQKaT%Zh7
zkL6BjhS-nh3~Ulk^_ONBK@8IwZ#*snE_AG~3*n{s$;1^mxjJXbZA@uNLo65dqOoxY0o7t1ne}JE{V^?D}&TU91o;24aXkoxdl%7Y9K=9O8tBUl(GI6lK~|y
zT_7LgTJP&h@4K&!0uG;y(<4uauZacMB3KU|NPuV~{~bo>^Cz$aMVSC^?~ojRcWGuh
z4Nlo}2P5V6qEcTJ3-OD0eOZNSR}!fqkg5orj?ou>eOdzs7Y1O5xO()OwmbbiSL0(i
zGx4UMNLm@j8NR?s6HnwtgcCgt)RU3dmVR~nYd3<+@4VM#f^JrjSb)U-3$^zO(&38g
z;vU6mh<_Du2XSAr=E4KPmN)rgr7-X+$xH$4w?PgU8_w;ts61~_!HK|n(4?D=9m+{sge~`Msid#bFSfKyDrsdzn6`Dnraa^&5>H2U=L6rgpDd+<^I*aM0AgfAoUR5iR&PO40SxBzym@nL4s6l?
z0q9x3<7nE2Aj#_v^^O&UwBe&$yJfzu-@}I||M7Sb3*)3}VJ?<(!w!ss4AgGLg@OL
zf1$k=%((sQRtN*&-e0jb&#RQq-9+gNRAm2+Nr+51%DUWQWUCJOv(x1rbx5Ral>D$c
zR7=(PSQK(mn`W>8k^~c~dfV*07xVM;6Y;M70MY%u)L$H203Iv0v1%0i0E7Y3GFTAm
zgTH?L8feKwEvmh|jET?+kDg0q<1iCTO5PM{!2+%aY*DI(2_lwQ&px<;?V+eK!8)WN
z4Su%Iq}UEIM$<_KYb9_kd_+!oV>_^C7kWP)c`6({NcPXvK$FI!FAD?jFlW}|Xp)X~
z(_I_|49UYRw8KUTWY;6z47W6WG41eXX+BS2=1?T(GiZ)cR921;SENA8L=~4`e}>XW
z?feX4&cH?)N(QMvuma1bChDe0t1OG`vXO@Uw1}Vt?FgqoZbrPAICapQ*x9b`=pIeU
z_E??YzK;q<^vS@s1O$n&f}l#Z;@n^0&pmtHfW{YO7lyO7smBl>M$w@ai~$RWuuQKJ
z=9V`mISWa;_0qM&C_%nB%M@j|-U%rVU28SG8Qp)*3%=YH4$Qg(z0X9G!XAlb$l-t8
z*L(XwA=UgDbF82WU=}@U=>NTftl^q0?7_BBe|iI~p%9|U>qh3Xne{JEIR3CK4Y?}?A7`LcIvlVC&~k1U_P~}|a2{*s@B4w<1FkdG
zVDB_S<)1uRS$~W2JA#D)v(C00vT;%lfalH}vZW4r`NSAp7UfCyjf|wP|EL8H5+b{G
zw(o_Sj2Ns9=s(ZKl-g&2iq%(SKihGP@S_X|K+0fjk*SD{I8;$heQb0HKFrT&6Kc%p
z$T4Ss#zcqJMGUV+k)PBk;xb7N@H&KH4Y$^&rHG|=?@Ljdx#%)(4`yi)r=hWGY^M1F
zI{In2+pzDT1b-L2=A`JY=}>Hpu?!5FCzdEr_Kg92;>A8jE3T`xE|sQsc{Q@>>^}dF
z%^ikGV)`X*o|uHBTZGsN$u@rYiJU1&1$bb?PvTwrN^eMz`7E((5&0~0@nxHt!|>1T
zwytNA>Kn=b@4il}jv-7AGdS<3{j{H;q
zf1xmx+;rwh_}>W(`ki<4h`aRvKwmKGlyO7hxp3=rA#%}1Om8IPZ@ixTa_QCUGGtNi
zY8nQ%kNJlJAg&XW$t!7I@}4(EUo0cvC-ql3THB@<3bt@T@sC!G9
zR5g5vw&RfV0_;Yx4UBICN?$eq{2Y~WbSWHS@j3onyjsfU=lkp9hc6Z7<(1%V2r=M{
z5AK{hpS&Zc7v>6NMp7mr#83}A^Mbd~a+4|Sv^!8IQ9tUBO$3fL-ZrDLg6gsRp+-Ti
zU=Sg=b{p}`p$-26K&!XYWSOliP!_Q@&3sPxVo`z|*j=j_`y+x*%C#Uxh}b`dT4w={
z)moR~9+*dNgOXQgdQf9!C=QV#c8K!R2qZgm^aP-Cf6VseOtybrqb~kL!#&kwbn!3D5%l6eK$APVE`gDgF?7dM@(5V?`2qy$uLc1C#!x
z!*m^D(u}WB-cKcC;c8-x;ePh8;o~hod%-dhV4;}
zVBzx?bOi<_z0cT)ZPCZ+od%6L^IVx{%ED;jfHclyXM|@6$;Os|9r!t_%1$-|C3RCE
zsBr9}LELFucG9f`OyNK_2TT@XkOH1ACyny(Ie{aDvmpp^~MNLu42p=JVu>^@Z69E?$
zjsSEPZx)`e8b#`C+2`oH|$Cmlr%Z=ouI
zpE-X7R?nW60#h{G=p9<3BIvouWMcB7g*rZB`uyy-{zrxP()1~;Xa2n!AOw*gg})`>
zXS$^I^dT7e1%PF;U#S5nK3Bx&%mXl2MnDwG0Om=8H4b>TwY=2It#1{Xj^Iw$3%Ma#
z>{^A2XeN7Kb8!V}@gR`kyEcn-;h~%OG>mh2nK5_Ckxbl8c|UN^FBlY96>XdC9|PN2
zy#He{H_`N%nQN&T|4(~g9#-?ZzQ0Uqn|8a{B~rGP5M@)Os3f#vSJHspAhz9v1}Q_6
zC_-W-nNq1VNQzP#QDjbYG?&sGnuk@t`&pd*-TUn2{Lc4x{`g(r>-wJST-R}Etv>5r
z@B6&Zb3gZeKdSP`mE0gf^q+7KLi;>nb#XZY9D{%;FNCBdQ~3mS&kV~2#gC7k#PwrlZk^SF=DgzZ8I
zU`OwYfESWY*?LV-XG){MycmwB2|lKg3J}ru4LAbeQ#FJCR03jvO{K5`nrLJpOLRx<
zmA6Kc`1p5!a3A&HCn^BJTAG7g0exF(?~wW^+|
zp1B=Q#EUcYfiZq7c^66QMZRCOt+uXi+Kd_2JHa$5%n!WZRHk?$08-T|_Jz3})(8672I8_wj5Ga;&
znz9EyGp?S6J|9IMOR7>9D@gwgS|0pbBv~LepUbI>zkmIWj=dDgBIKp!X
z%~T6PP+rAX%_Zoki`Dz1jB~SPd
z2uuGt<>`OcJDhVQs|;HLU81-$6SUU=PrKW$rc!Tq1Y-Zm(<|ez*dr26_cGhJ7XS3%
zi}T9=10C#-wc-9+cR#>a8cT~wHq+UI{UBn9(gfc8rOX3Ol#76EYdMoUk5I03n
zEpmJWq#(qYV225t99RvKIRzKxfyOzgRX4EZ;B`#!J3TdI7INT7v+^OYh(TprMSlqv
zPd9vwG}*&YGDd^*Krbg2u?w8r*HqL0nb5G9V*A;Lx{CQ&`uYEMl8vK
zI?MpV9Qugg3C*fmv%>$heh?pc<_*fy3zwg<%_DGfuw{7)roG;sH7UyeWsU)pwZ|&&^
zXjvxUl=Rtw)uuzz{risj;m&J3C7xzF6e>I1KcHr#*~xLOS+*m*f8fzzxSXlVqjcG?
zSxhy5jaUb_lIR=vZ^ux)7L@??cF+xqfThEH#C4j*0Srtgy-OF@tNn-qORn&F8lRqJ_kRP
zkXc?2@E#KD@1p?uIG;T~2L@GMW|8=vjsb!?ASgh`K}!$x+A@GVpeJSNfB^5I6SxX*
z1Yi$Y@=ZXJ)YIZ&tzJOw-+GTXbU=g-Cg(<7LT
zX1L&p$=rW02vgde_B*5T4jSyL?SsH~n!rXX0^0jq$lsGxs$np;fOZu&7XHZp=Dq*F(&~IaYn+%4(I5-u~JCpuPC*%3$L%8h`?~#
zri#PgD|QgCLI2AVEi-Vp*kMq;PoAlu12I8;tZ=)_+f27iV)J93$0gqi$Fe;sK+DnF
zBu=?C(B<%AZ_K1XWg<{)Izi7MIa5XRU9w#$v#Iu!tW2j7Xj9Z7L?`va!B|uqhibj0
zkU32IDB2c>07i-%lrqJ0d{^o=-$$BejP~FHrwX#{2G%ph!Ta7v&Th;};501H%p*E|kcWaK*gyg4>oLx5xi{XJkK86ByThiV@ieC%n%vnQCfw#vI@u|B+!tKGOPWM^>BOO%la6n5~
zl7%@e;WvV@krqiLzKu!?8V6*D%ra0fu*0NdlhuGKyaPZCZl@%R7)>y3xgR%4g=B{0
z;0s5vP5AQV#H-Nboo;{(Y!->^k(L6oDgrLojliwJMo%*gC3M-QL#q9ivXX&lOH(_Q
zoFKsIFV(x0{&nt)hT;#^_yo76NgI&I4|SwKoSW-1-2Z{hJJ4KG!bT(?d*@8j{dNTC
zm@8#1mIH2zztlq`6g>n|wa@@07cS(lGF0=uA{t`kL*#-{CIHt!{w_wFB&$bms&7u{
z>@{8%aHfL43ORW0N5d+L5*pkva7ThI9NC1o!AB5`oC0Vwn0N_{raX=}c77?a)bwkmM
zxkCf~h(NeLy0Om*Bc!Vr>oP>bLDz}kz-0F_S+Ot1PDJiAqk1G9uYi3mR+%C}O48UhFz=y+z4cAq=PIM4R
z3=?3CiN>_d!ib^YckN;k{_dgcn+cT&I}E714(Ms2{RiMZ2F`5=+zsfvM-VJP?Km^d
z|z>eGo&Mr|8af>cM^VC@eeJ^q@A>qB72-GaZA)YCI_&%k?NG4GzpK
z@(^K*;KVOs=(Amdq4Wll4JHf7zG-;wsQc9(6XmwU1rKCPE^#!u?u_aMi&IFp3=ig1
zeArl`G{c!|#!Wa3B{lth7)BID-x7)b=^;R$pFDj@_Rqq8TR_2OHi7K8EP{xxxJLW$A~VAVrFLB)b-%^gL{Ofc!aWHd@@AMbXZo
zp)hnYa)=D#OcIL={!sic7_CY!)rD^bh|3rzRA9il-}BCztWhDwToO0#T@3O=0)41y
zMba?Zr^E`R1BfmYE?@&v1Xb)WiNa4h2F$R{80o7)(3hWlN9pmm(B`-|2k>)G2Mux758x}84^
z54`?v@^b1iVH&~LhEgQZHLO8TY7AWdbL!^HcNbZ@_SKAM=_0S=C|4nm)P3Q31(%pM
zY`1h=U0tIlK7~S7Q}=#{Se0i8tgol`{j|m%I6L+-jX@$sXH$MSk;;kuOLrx@pjHsH
z?M?4CAQt)VT;p;`E}6*AqYCRJvwSjx>kixW7k|Gzqw4#J*I^$&8E>>f!@EH^flY1*
zE!Zach!HMogI+mw7w%MJJhtG?qzO2uC4*B(Of?WU{&x67gCsE9|(V}++p
zV=Xvbzb&Z@GXNIdE@|nFykV2$0_yM)*?c|}Y=v0D1WdR1<>KPiWS0OEQ7q2oTjOmS
zpE=Gb`}rL4dv
zy){)cl#Pz!qTIs_OxCnUUJ>0Tbv43fK$+oe=+={iYn|*h=z^lc00=PuppCFgRV4O)
zz`@tc+mfy~W$JU@thfUCJyB_K9UFopM-osnQ6MEmMY!ioQ+K6`DQgd!ss^?Z4~>exKUf+Q&U293odP1kzI$-y{ouGQMNvD6PyK#m*{qre%O5+TkF?BCa2w_CEGb_-fBN|GIkO7n
zON>JCWs#kP+jzQE3cB>lwDAKa5p1qUdIuppX
z;n@Z~D>9>QKK6-QJMBA&7s!31?g76)C|*$eK%y$I87g
zpIwN=R5zDf{GwpEBcV6lnT97JM+1>73hb5^ypN1iW;Rka7t;Ta>thbdX1^jJE*5(<6Ez?D*q%KO^QQz2T$26|YzLrnwMZz4_^wTyA-
zVr;ioo!idRPx(;!7Q@R7sGDGL+r+rRsH&^0%f6&jtezwxsx1>Gzjm{`$!FSau&k;J
z*AHEoaj6S`9l}!f4)&p@;F#>*1%?g#xypKlndOkJBSXFX^G>gW$~PT;nhC8bf@;Fc
zRc!bL$)TXh(YEZC!eK%(nv{Cl-2$Hj7&>X+X@@spF5m{_{Y7L@N4*pbh1mr8Jd&VS
zq1SOB&1Sg&nrIdN^5sT2sII1e1sR*#N7$+t*T2Zc7r!KPr*p}%Pp`81mje+Z65=)I
z&n$eHr&p(g^tXq`fuLnanjX-WUR?cs2}XMbB4u5sCw8E`k742=*)X$x2*+t)PAH5R7i?0`As9d_~%Jk
zE2XyNZ*|L5a6^JfN!wSe8tw3s&WheHFl5Z{3orc6oPcnQ<$N_=QujcHiWYX=^$EVt
z=t2dgJWt6*b`ni{7F=Hg+`VJAUT-5BBHam)&bVqXI;(Z8mMT5evt~+rHctpTyoXN>
zMGz5rJ5BB(L#Dg&+)
z{rmTkpy-!w+|&I5&qq52qxP&{M}Lxn6)@Eehk9BquZfGDDXo%ZC>5j6|Hb%&7*i+~
zUv(^R0_}`kT3?n;EQcl_rR`ps$2RF!CtMyFB(cC);nm_v9Iyu1dJR1?T8@AQ(ezqh
zwV^C&OpDUMVt_F~hK!GYFMh#H#4Nqebpa|z$Ylv&qRJoXGr(*3hd%0QeiHGyE$sNd
zr%4`WBCfX+&jtnEM(l=d8ELca3OmXmmZ9k<;J^@&0?)9@pjJCNsp|Bm
zDE^^(KnF)30er&kiX$Iaqs&FkgL9CTw*In1?Tmadm9|j*0)q`y?3%_pHE1go!E3#r
zt8A){QD-r`g$%7LwwAMIzNsQ`B;ZG<^UkL#NBU~&@l&1
zqhTL8mTGGol1k~7e2GGHx#-_t4y^vMD2CW9nY0G`HW5gLpbP-}F_L-?(;k0Luqx8@
zUDdN?i|JL*jK2T+rSvN9v&8>sxT9CrXh4mz?mpHLM?hakCm0?HG|JLgR~6qS5?OVq
zA8B4H>BDcvrf6bk=F9>ak5K6c9W4J2|A&dTHEN*{-9Hf@{y2V)
zlvIRn%ln1~QD5bO$UKx1b?9ABw)O1!!tw<8^_R7dDM3^aBrwqvWv
z;}#n`8kq(;-aKEKi4X=^TEfdZsATd%aO}dqK3=~z-Z|+QH58FRkVfB$pdKlQm?Z%d8lLP32TZkfm%J_JGqn~WxfN3UF8l!H?k6V4>
z0~ACHJIm?AVvSf_6I2j#HQ5AVks^qA(_rhILT*jZfRq%i#At`rCjdXNbgitcf;I0>
zWJ?%Hm(@>8-G1{+w$7Y%5vEyXNo*FH7>v^!4EX=T@
zD>PHr82+|@zSX7TTNb<06Ls_i8Yi4N@i^QdKsDKNtUCWsP8t%&Hn2N6oQJ+kbxfqj
zvh39|a&smv_5QH$b^lc<;SYfwE2?ZPqnK}1vQcASaA+;;bL;3+9U}a%u3gX^iP1^*
zpNO-;{XT)$*=atVKp{P4nknA=GSg$Y)rF*Zz32`PA2M*
zJJ}NT0(s#O%i*!IcpY4t$aaiIc#?$%U%A4^f$jnES`oa+dPx4+i`9Ws{`^hQQ>nfZabFXelCllfasjAZ
z!hv=gL!Lc=gn+o7tbCNG`=3b$T(!?ce%6QwFT&KQo2Z)ag(Pw5)aIUk+dXyA>81TZ
z&>8aVqR;g&c4l<|C9^zk+Eltfi%ByVC*L!qUU8)K%u!R~u?kd?;Rc{qiK0`x^Yd_D
ztu9EoBk$bNMbc@(wh8Qq9q^&zl4ZaYOvT!6dvC%l35P;4QbfViDGdrRVo>KN3N`>zxF$8I%ieZ-i#18S{2r;JW9Nl82Tg)q@kXIr^EX#q%C
z_`y+h^0!{5M)^5TaRMxJ)`o5(TN*;P@W?{xiR4x!S#RH3efOJXUAN<#w_{_wBLK;S=
z5DYc&RvvCiijH1DRxso~4U*>N#yurU-&mE?7f+s?Kt&M|eiQI9Sh_%QsDP#p1Br5k
z(z3iL|0InYjH^awIK4sz8iMLG3fSP}ji
z8vKLMSndc2OsTL!nji0H2CA+#4$XI!WIw>TD#o|n0%g}B09jx~T`es`Poq8N%H9+L
zPWK9SbaNT@m7~RU(fT0HeHWm(Y#Dc#p<{-yP|^WfFPN6~B=q`qH6SezVqJl0#kM-~
zA7B0^R`8#$6xy&R7b$kW0Vb|}|9xI>M(-+P5a&bE-)~xpOQ0A^
znP(@tat^WbkvNg#shp8M4B-4gSzu&_|wCVbOB-+8!(aFkfy
zm5euAy4%VO**gNf9{v1}TltSO@9^nSsplw>6FUJNkS`RNcS7m@>v;cmruN0fW@^`$
zAIYBRp$ZETl4R1DCo#yqkZ@_TSeE(bM{cs1f~04odG{8$c*)j|3K)I;X}yLbmC9;=
zxtqDyrR-_r{}ed66_mARWdHK%qlvtlP1keGln%uZ^!_UlQ38uZqOAGVn~>Rkscb
z7yO5sNe;j@tr0=sL}wWsND@BP3Nyx(5U62T?7f*BM@h+|c3)MdAmPDG)=010h@s74
z;_tZ+JYvPEMG_UJ5Jz%p3fS%yS}AUulTf7vHc5?zJ?3WZZ*J0{QBy*IlK%>_TLA$>
z=g+?znZy+JX-3;FgLISB=AY$Ms&MR6!pFqbJilOt(KFQmTKb0-xc2mRKz=n}(Axqq
zQg{5Vbss=aJUu8BkHw|PMY8af;AUo~OZftirJ~BViv}hPoPdx0g-z
z4+v;Me*$B|ZvbU^wsbf|ZS&f{Kg{i@{n#cCwmt<(8I9NLr~wJR@UGMfQP
zhz+J>SQ2V#J}~{r;-{}(`7?3n&5NI5`5TXuEgvr9%%_lz94SiOCWLiD{U>%98gO$D
zNcl|Uv2kw{dX7>!1{bgg=gJkS6Ufr##zfw6U~@ArYBgZ78@hhHe9S^F8d-rAnptRo
z-cDQcP|0d0uP1tpTSIlGxxH}$&?1trcBh3tIl^;c7c<6~hXDB^$4^R-kP0r$E3=g2
z^|pyJM&m_J%n|h|K5&>R=p85^v607*QKb7KrJtHSnLLRW<+_#}4bDY{Z
z*@-eB_}Q1V9}
zs~tz*zv|4@0*%-$#i-||daDysPjX}+G7B(w9athJAUGqHL2sm)+6=n71tWT+c)XjA
z<9CcL(9DsgqlZ#^@|=0n(s!*vx;}~{qeMkN`e^wNlr~P{b@#T~pYiQb<&Z}OlLh>N
zKDeR(eO&&)sE-EbV=P#2q!R?%h_eBuX*jg#SUf+KCPYL#*ptO
z8c24}_uy74LBu97er)T)cwWCTYNE{%KC8uGfUO!C>WuGDLhA@s9$BqItiGW&vkKG{
z^lX3o>E1;F9=E(&i6ks(Of!{fKhj1;yJ9?wl@(1r@bct?xb^g{ezatS5(>O{giKt}
zY^jYG7thXHqwq!qo$!4G@zG#t8dVR2gU_raQ2SfR7>llcoGViBqc2r%q8Y&D73GE=D|?Ko%ii;p#5YxCTn0jbteBv$VXt
ze4cE$7QrNcO33KBkWHZQX>*_
z^1PQ~&I5|J?mE@XEnohyow%?NfsFn~WLEg@M_bX=0v71>yo0Sobab?pQ5n){zIXax
zYwy2g;uAM>qc4sRO;1NrZIw7|q&RNqEw2DUHH+#@l>Y^poU%+7n>aJ@0V5L#L2u>(
zb8p_ePQ?9)0%??4`L~iPqrp8~*5t1%10Wf0eB8qr88t*@T_|#L`1r<$r$a$cJ@6#cUo_Nd67B0eXNtYvCK!$v7?Nzj%>Yp`{sN
z>=tV(eCr;st$d5Sm4Weh=Z*~J#&^hq5Qs(}({{bj7@o^V%W{#lS(fOor$DOW&wgYo
z>@D-@Z=V~NxzxkawezU)4Bp1vf4$UCpFTayq*<*Gl`M^TJt_Cltw$u2q(Xk&;J^gZVI|Et52Ceqjed5Qyg=%~moqTLGmhP{|
z?*C^Vggy1YdC>pnLH}_cgsX!#wZzn~X~GqT=qVJMRF#nPGYq@z-*p2Zy+(~-6kNZc
zw4WT8mCfs_=#iN@if=emf}i;B#7!|YIk@XqdjVoGQ326QitZkloy+T*Yowf*94d)a
zN)<8scO}ir2;k%~NnX`N_37kUP|h;OWpfesY8V#L({^}WZ%ajLFRRzaibu@KBg|wg
zrWY}Z3S{n(2>?lhP1J=CyfX{#3*M5W+5GuUFNe3yG|dFkp;)zfr|RDeRA0I)Hv->b
zMny+ApQcWPOFrj{D35E3vdJA7&Vxi9B*WqDu5gED2Ic_Lrb{Me5z&y_I
z2e;w%`Imw=RvYoCd%rvch}qCOk#@RdU}^{Xqfv^Z!Mu*9VbQnNG3-5b1RJ^WQ$k)|
zAJu6w4X{CGPLtsXD*zs(z4b7FPXc1m^?)M=$PFp2cV37ddNJAK?@9hF<1h_@v8%5y
zA0_uAG8DiS;QeW}W|Gx;6*T7lurtFDV46(}f2q?I%4pl5{wH2pf-Hk|75DjHn|3`XL4b|X47Z_XkEBZFY91K8hAPEmDZ2T9qM>KGK<_BSBHn3$W+Z;$5Gcd%6@#eR2
ztAk0#1#FmIi2gW&L2EOwSH}HURc6
znO?ET%}yBvJf$H;Qa!@lORTpCDqFS8zQ?j$br5Lu)Z~E239(Vo=81WdV*+j!3WxLX
z(ZQ$GB!XhFTYrEXO^~A<2X%rK1e7P&g-TT<@yyQy@1U
z%{mfRO_T?}oS4|yDVXU}Q(BioW9uHmaH>{$9mWyvZUg+{VQA_i-@X$>31(z;^aKwH
zeuG$}=_t6$_9GBa$N&d?+bVOwk&A)TCFfr9^Ao}w+^AGagSBpaVvgy~m4J;VGsh)C
zDq(pT4p=Jf*We=PK~$b6A3$Z&SnS;QG1>QAK!ET}WoHEuISU<$`N?RYT05^~sMbCF
zVLLrC_?&B&$tJzR)hv7UGKdS>>AdvV%KGyJBC8s0QeJu$@yp*c|LThIOVQ$E9
zOXlzfWYe1wwt$WAE+MWR`Zp0S$}E~lvLCIU$LYlSL|Ikq!;ysW$~%
z%+PqhzE6M*D(NuLys<}!R*cz{c5X1A^AW}&31%Uav}tWY@q*aV8YA9m<2HC@7|k1{
z9uE2uiSU+1&4#WlF>VXcT@Xm+pkKVI?N>i>=&uto%MJo&L!5-xV@Bxq!goD>1I
z2$doL-2lG1VrIng_zyvHaB)OMBj7%?M7xla{f=fLS-Ase89~1e_5$r*fCP{V7gZoEO+q%zfLJ)|*4Nm;QVqZfnBlju?rYg|e_5`RGc-0$Q8{3FPS68Qu
zg`|Q#y)8R5vWnC0)a^Uc=j&KOoD8fQD<5PiYuDLqoNghd{J~z*&;#sN?%{ycw=cg9
zPZ_%s@1Y1kht@slF#8zSg5K45zD#<-0^%lWu+eD^fgM0;4hsw~GFe!U`3!PQgn3mz
zhuUASX*(^eeb$Yp)8f`43iA!epXVE;qjCu2R>Gd+T^}?;l~{8ZxAy5H7&_a
z+zBT6@Mhpxw+b-MYTz=#N}(U?>(X|nCy>C^#~I4QTk|i^6i;BnKxU0y3Eb6k(%bs-
zHnXfi^7F}ca<0Dp>g>UxrCIW;Y%r<{FeBvaUw)Y$PAn#8&1G{Uq>ii&m?6?GQf%W5j@Y5erI
zRf*qY={D1VOeu!D*F)Tk7RWu=a#n@&U<%&@Kc$Z`vpTZp#Nmct*)|~LtGL{zawz6n
zYApICg$E8}=Yn7?$-@3tZn~G!YFo=Vm@>)GOgU@XQnHg3Lw
zNE|p%k@kzNCnHmx;K2I>^2%82@0E97U`wo`f`w-pD4ViM$cOiRcqXn%K|G96M2
zavf-5N$tYDSctd@af7olOrcU>bEc&zxBz|kUrxHLg0mELvS~ujnaNh@@eJVcBjF9A
zsp!ZGDG0P%u?Z~AC>0F3j}NUaDk{R)kcj3gsPFTdN~cG|KgZ#@OG|VX9Yai-W#v=f
zjHEH>`{6BXf$mTuBM;7yqI13cVf5B_n)YHg99H)ZV!EW5bre0hdsG^!ZJtd8t69X=2gyr%!dztHgitaS0p1TzjJCDjR^K>Hu`Eg+2ijUZriTw@?kvIk{Esf3YYDp#KJ}9dJOJLWZk!Zc}OEUd*)J+hJ-b
z2FhjE3)5jl`lsYE8*I8<)d)t8TbKXQ!_Mbrl&N$zjUn_ZQW`5#BpJ{M~f4eTI=dFDsRI
znse?1W&EH8D7RmxUxrQ@*3IJVs!@#QDcqzH%n0BsvgD_C2!7#Y+J6(p4bA4kx{`MZ
zX)?9l2K<1xaj8zy(kkjqr?q>TjTeY#;+!8q)=1$GcB*B5zJXT0`RkNbN^-L?pa;;5XFfhx=U%
zCc_(`onf3YR6vj&8EFyYhO}bT;sv;d2+)EJh?{s*sM6i#j9|x~9$qr(ngXK3K4`gP
z(UW44q3LxnPM8PQGZ=C!&-}}r{Vy2nM6Pp{#8idRUo2Pt&Ak85@9_s<^xrET@D`6o
Z##~lu$XR{XvxaI4)>q^^zkJNSr<|iwS;!!
zs2oMj${_#EpNm(<%9s1%uelebjvvL}_!&Kt?2Nw_n4VCzps0&)opfH#^-8Yb}gG!<5KgxN>=v*lO1;YUkdSiG7X`
zkLs9Hl_VFjc6Hd>9YL#?>+7s&ymsE=&;qsHUY;A7Y4eY4dpbI~!ea5>1>bUoEVc>7
zv^4(sWTKlt>Ll*8gri5OzF#z2f_?dN#?|FLtL|oI{B~zH^Iq~x#)|nFj2DixQ?tk~
z8?I7|$={~`FCK8AlRLv;=_avDR~8JOJbhZ}!iAR{VL?Gh*Q{ByUwQRLIXSr}4xt~#+$YstO6Hss-U*zwG`Lngi-~Wdk@cFwsMFor=
zG;MHEiqVMrbbA@szJ0PAg)PrmTfY}!bFY1cyPUTf(_**9?_adkiQhSA-x+!^U(GP+
zd1q8{dbq~IJ;}PuP~HmD$Uud83HADOn0&~9Z~cTMmvZ#tv8m^x_5&HSmW954z4xOB
zP8PC-oNS*_`EHCntyopr*+@ZAv8^q{Ux`MeaY$hTEKz>yry%9U?UB1Tz%n{u|ES%X-EBJ1S|qvA`PbJK_jch6
zN7yhjT1iiytd?@0M^W?VF#hRte7(RgwuY)Ke4ol)O!vLDqb>e1nNr%XH`W^HuAre2%_eHs
z`sg^DfvNbsWo)Im%fU)L|F(t?*M)mh`k(mvMs!3cpT_Cm_NG&mkJtOl)rQTvPj~Iu
zQ5by6*Rr?a5a;&o9Y62!l)Es^vaGXhsZ`&BZ+xh`#%0l>MS~-a?*4;G^*ttqqgN#)
zdeX+D@nMUqj_I+AbbjY4hdh6A&3kM2$Vp4b`s}A(Kgnt=QdCxdb^g<>o!s1~=?ZFU
zk>TOHe7W$PPM1W|pJg@58XHHC4R(ZZt0(DKM6274wS}yAnEH5gk-(L1wM^XkbNg;t
ze=$d_=Kw6)uG(Z}jN`l4c8qo5;iE`(SH_0e4Rsngq&iPe&Gok*={=w4cTkzmZ_zCM
z&bYS9hG+kNrLOAuiYGhebGh)0DQdawrE7zJHy5#-q&xk7?Eh@76NZ+e@@6rqlnqa%
zO;3%}gYca4n+?g!l#;%ZRKO9di4ikJ1%(szv+>4JFJEriqn2=nOdE6ax7_Iy=1Vq-
zsj+A}yvg%m#KXhKC{N4I@B3(SIi7n4r^b6aGFdeF>-z0d&!*bFBg?~o;@9@1uJ_gL
zT}EZ$iXncQ=gyw>sk3QZQqunZTJ_}QOBc+tIm}G!Bkr=jYvB|S@UR_S+P_k*%a$f3
zrLB=>(VP-+7XOfuiGM?Hm9y92<>kE;$&q9u!WLNj^4OyFTz9zz>Up@m?%9TAHkq%R
z*40Q;=HoGr_qAQ{v9XW8!*(+zC2ZCHgrf~J%dRhzMZo;miKeEeot&J8M*|
z+AW=NL{(@gvO3FlBBV%r!TEsMRP<-^83TzT3TfB?v>7^^U^^AFjg05%o_L9lm%6U(AY+)g1R|*2VfpIEjM6nKPSeFlJ}l$6AG(CwmhLEzDnZCD
z#cUjGFO%9TdH+nZb=U4N1Tl)zN<~mvwQ^&wqCX5gAy_wBQe)%6mr^8=;wfv+M6+OzM)gxX>
zc^Q@jpQ?&C@dpeMDyV*=eOr-}aqbPdbV^M@A!urH+^9ayQOR-{Ij=YBwksqmhb*Ycga?4x#w(v@wmm^3ve_Xp;3{qvDw}0p(HWd_@^3m~y)?
zwoRqKr9iL>33=%c7V?4RYowQD96NSQ^5FLES)JXCub+RnjVa-B^?S+c#y;;_QF0iy
zd7mid&L&!=@8#*~^T%s4&qXTrW7HCh|NN|avUPSrffq-uSr#Bz>%+<;jB^y~2aC@&
zzzwxm$HzQ(_)QZXC_IwX`*D$gCIN2{ZZlC-{I-ObV%o#0fue|I;PnYT_Ye#M`uJ#VMCwwheZH3lb
ztlZH5JC$$R>5yq|UZwYA@)iI7@##8a~*0XaLIfoNQp
z;<>rHxp^pv@zVT^qop!uj(*
zREPO5*CfbZskQPt`~FhcXn)H&MaH*M^xbEf{rvr7WCJDrhN=o*94U*{lR;{`T>17G
z;LPLt9_3@lZW^cbNz#hHztN7`0Vs^0{#m9p+TAbr*J(RAR$q>9yUC()=H$t2gDej(
zua1iJsdrN&S>h+@#%ZIvOV~tI)~#Eo>&3V_mr<#Jdb^H2jj}sXG*m>Y#^x_fr$2e}
zBo5YBPSo!Pp$e%8;PXVY2D_nJHQ&mbwT#=5
zpdV9qF;X+jTr<}DX_=yuQgEvMa64`gV8*a4+=Dxzff+DDMM#8k>RWfztWVKsvMu?3
ziCK(*rL|tlQ-U!MmtX$R<^JYh_Ac?4tAi!j*=g-n?~Dp7j>sm@1o_;R=2UlGYBFka
z4!^ZFM(J#>rXTqj-(hur0YCXAW5wY#<`l>Ap#Wv04!ezv_YU6N;KGyC`cg7=DfzdH
zA1Mv-V|wFZ3{}xobbJ%wl?O*#U)HAh&Y3B|`e}fgD3zC4mn_on>J>8%!WQF?lM>vO!WCjvRIFRMq%@o^{P
zk-G?2rp-PZzBGH2)5r^;?u%C#!{(y9E0QbY-1|#HMhh(Cjcb1mk8{#cRd{4IF0hM6
zn82ET`}XY4p3GQb?3ZNUEMU|=loH?MLQ{1N!7eVY0$a5U7cR^WXRNCOZaPc{me1GE^AWHc
zEK`1Wsgl>j({umG%p}OwrBU2__LOB@o%Qbgr@34+-y2XQoj#9z2q<211l3tQ3c{`N
zUuF|jSI&j=H7Xm90J_Mr&Q>qER_)(JsMwQ
zes6^}r$MgwKA-)UeWr$;rLycB+t9>Pz_4@&jb71F)>#>w_s8-UwIBMc
z3nf=zmwDoS>n`ez_ZgS`NAyA%oO)9vhO(iEjrZc6B8_-szoNQ0gU3bPNXnH>
zj1gr2c8wCz%Af+(RaErjP?B7Z+kVO?h*Qc_Kht$hi0GL9&(C-Nc%G)Hq>G|cBP=46
zq6V<6rNYzZ*?hjUB57)@!wN+)t?ZyA!dm$s4<`TX*M}7y9(WcLqx~}WJ#VtuN>C@*
zl}oBNX1Q0~TC%>~QJ2B$Q9tgyd08?tG7=Dztj0Y#(IQ?k(${DOz5pe0>qLJ+Our4K
z?>aRYDCgtUDy|X5LAVhqUhJlMu
z+=2s7tlj-@&SP^a5TAC?(bYA~ezNoT_@tZgc0Rt6e=q#Q%GRGhF940?Jrj~2AL=S$
z1ULEVy#g(aC#SVW%1eSrBSmF$d{}V*b;jh*;P#x`K)!a9^TZpw!S+|U!E-{Z8OIO4
z<)<|5yWUm3)wyTgRrM-bJ=t)2YEo(GOtAFZT|Y;7`}X72@g`n)RGeOFFA)>YigGYM
ztataT_CxNJ-ouyc)6`W}RkQQ+mCm0(&v9!e_Ah_aLh0$~v|(~qplAvfa~vbX_lT8&
zQeD2z(W0HC(TcJ*IY(ZdHLsmyf%PUSE57|;T=i1jF$v<-7RMV!tJ{@lqZLKR$!txsR|__(n!hqc&`NDHUL3k*GFS+v5~sP!e3J
zID_X70Yar9(u3KWo*a%ia(n67IQ>VJuVxkpqJRPL8b>^2Dr?Y>W
zwkXbT;h$WTr6$e>5HAm?NeI?monwqchZYD?CiQ8etz}h78t$|4glcNkkQrsvlPrrc
z#sdQ-GPg@BP)H-%X&U_c`UrSgFp+Vy5<&mw9JU3B3-V>U2~s|M
z_*(7E8kasyMQ3&lwmVO?IF|+gmrVe`u*h+%nsH#q4H{z
z^vgQ5K@t<22JNg+ncu9*vhPU^!B_8ZwFe0G5j4^G>UjKc7iWD~SlIi^`fD!(SD!u0
z+W*^+FJ1FYUf0xAZ04_D7qPyoR89?qPEOLFKR4FqEP6FGEF~#<%gy7cS5<%U-xSij
zmYeD42yw(9l3q3Us5duRg5a=De(J?8!X^{@oJESc7gxuB;q!b)(06WP717j0o?yK`
zYJMMHos%l*-ry-t9)GZ5!I*}kVwr(>!b`uLrk`xB?11Gy#lzMdiM7Vf+vYcO0QUc<
zKMBnuJb+Jc-}0(kSLz5mjJ)nU3(6$Pvh5|voY>oZHw#)y5QM{qx~h!|0}h=ZT(fFb
zAc@mGoZzu0Jw8=^;I1-qW7HH)HOyBji~fyin%A^7{_`KBll?36Gk28C>fJQeJ8$a*
zs8J;ps0v=(>MzG!E6=>y%^0%5V+2Jm%Gs)KlbqU-a!N2Rs=9c
zv2z=VL|Z%ZXu@zyX{p|5P%ge{tWSzWb{%mirv%m+K
zY~EgcrMdTjOo{@JcR_@RyHqJBHyB=qNa-69rgC7X}$Y=%bz;RTr!0UesM4Mu3b%e
zYri{d#l?F0#(9BS|(Jm*79-DgQ0hg
zOJwcGZOD?5DA5^P(|{7J&}-VRj;9PE?n+PL*^1lh1S
zSd+;!2*|()7?yS4g&Wa{&8&kx^_qCL<+d$;SMpzdwzak0?CuMi85#D5gVSEr-U6vI
zj0^pxy3%FRDczG>Jsc~NRtzYy-htCQB9^>MaiO5ZySJ8Yif+pB@~stMytdZ)H-MrJ6g8<6aU*$oGvjrAeiVOoc%DURKW?qh
z?Pepdx;(^poO@@e69>ZV&)2Z14B~Qfa`OH-4k+PbTt(D5J?mk{8E@d5Y7#B3lq1|T!=BbrU7PY+KIeAjjl19#xFPy5!`a8FEJ9Dkx(v|WKo
zNw8bx3^StTL>bqor2<0s4c=X~-LgXcionPOlrd`>;|smh|IQQo7cs!st}*`$La~yO
z0KIEx($jf7B9ccQ4Rj~>gr}saaeGN26}GCzZYv&|IMoPz|{B|81XbS}MfOr62
zm!OQWr!7Lg!hj57@8?>f?tA&W*t-g|6FV)O*)u|&Y{#XAf2krkang4GnnN1R#)}j*kQN72}))imBebQiE049<)o~O
zh)z1$lRlj`Hq;dXfnnGF{pFo>#^?7sv$S5RcdSquo2W91@mu3$)Ndm~%1ut1rbFMH
z;b{gy-K}j$P334#U5JBTV3*&>llW_8;AHtZlc(x4r8a%{{0!hsfvQ3E$*3PFhKHb5
z{YlnT4flFJdGf4(1jr$9BMXb3wGiX`PrPNPmi@IiL!ZRvXIU*RE%QVflm^8IVz(!$
zgF8^vpqS%wT}ooUcKXC`DYs4Hbt862Xn&Miy}*l6>x~l*$qjCO1nn`**Eorg*`-&(xxrBO`kf7D(dEL_yNz1DK8g@R)<82#Nu4~nmM
zO0GM;VabLAXGTXyLBZk+y)R5njDFd6`AVH_2zT1Z1z(~|Q-vxdorMd=o=vh0E00tS
zbebG1vsLSP{5|axB=h-@
zwo3ycx9KO>cAmOB1{Q<0emjZ;$PUe}RDc0!be1)1)^$C5wjnlo9u=T!+xp^moNd1@
ztvJQD|H-3A=Pi!i(OtFubSVafi*`26K?t%v>$d@SA3$-VI8iqD%38*wS-ui@5xSL|
zwShs{pRfJ;_SkJG+k@-R#0(-bWMaaBR=n{_4fvy_PemCwM!D%i5m*)6o`#qDy-f(h05Ik*PuPMF1(@wlWL5(!kBY?b)2$_tZ8GN8JM_El+O><-VFE(Tp2)d4PG1C%0u~C8
zo^HL>;L9g}vZNOx32#fU=b`yvDv=sL(Rkhc=pH{EJSXJSm4l
z&aLd|GSFR_D}g{|r>Bb+x-Cd2GW5@{wsa1!900aet2Q27rssNhbm}-htYPa-8VP
zq6eXnC3PX2`|(xVr1vuSdhgTb*UkOw)KlJLUK(~`8cw4+r+oNnUTWrdE1LqIjt^Cv
zShbh#%*lccV}F(XP^T=eq)GcdY6QTOIEQDqAVyy*?b~xU?sX`c8|?gmYC;yxe|4?n
z(|tCyp!+E#vKTU+S8B{|Ctc}@o;7bNe}~Nap!SFhMUkPQC&0Bo_?nj(G}egO`3n68?IZ@E
zW<8!DOlGYU9ydyI+tj*P4T7}6
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Batch effects and false positives: a simulation +study
+Jitao david +Zhang
+ + + Source:vignettes/false_positives.Rmd
+ false_positives.RmdIn this document, we demonstrate the necessity of a proper experiment +design with a generative model which we use to simulate data with +“batch” effects. We show that a proper experiment design helps +experimentalists and analysts make correct inference about the quantity +of interest that is robust against randomness.
+#> ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
+#> ✔ dplyr 1.1.4 ✔ readr 2.1.5
+#> ✔ forcats 1.0.0 ✔ stringr 1.5.1
+#> ✔ ggplot2 3.5.1 ✔ tibble 3.2.1
+#> ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
+#> ✔ purrr 1.0.2
+#> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
+#> ✖ dplyr::filter() masks stats::filter()
+#> ✖ dplyr::lag() masks stats::lag()
+#> ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
+
+
+A simple case study about plate effect: the background +
+Assume we perform an experiment to test the effect of eleven drug +candidates under development on cell viability. To do so, we treat cells +in culture with a fixed concentration of each of the eleven candidates, +and we treat cells with DMSO (dimethyl sulfoxide) as a vehicle control, +since the drug candidates are all solved in DMSO solutions.
+To assess the effect with regard to the variability intrinsic to the +experiment setup, we measure the effect of each drug candidate (and +DMSO) in eight different batches of cells, which are comparable to each +other.
+In total, we have 96 samples: 11 drug candidates plus one DMSO +control, 8 samples each. The samples neatly fit into a 96-well +microtiter plate with 8 rows, and 12 columns.
+In order to avoid batch effects and to make the operation simple, all +operations and measurements are done by the same careful operator and +performed at the same time. The operator has two possibilities:
+-
+
What is the difference between the two variants? Option 2 apparently +involves more planning and labor than option 1. If manual instead of +robotic pipetting is involved, option 2 is likely error-prone. So why +bothering considering the later option?
+Randomization pays off when unwanted variance is large enough so that +it may distort our estimate of the quantity in which we are interested +in. In our example, the unwanted variance may come from a plate +effect: due to variances in temperature, humidity, and evaporation +between wells in the plate, cells may respond differently to even +the same treatment. Such plate effects are difficult to +judge practically because they are not known prior to the experiment, +unless a calibration study is performed where the cells in a microtiter +plate are indeed treated with the same condition and measurements are +performed in order to quantify the plate effect. However, it is simple +to simulate such plate effects in silico with a +generative model, and test the effect of randomization.
+For simplicity, we make following further assumptions:
+-
+
+set.seed(2307111)
+
+conditions <- c("DMSO", sprintf("Compound%02d", 1:11))
+# set up batch container
+bc <- BatchContainer$new(
+ dimensions = list(
+ row = 8, col = 12
+ )
+) |>
+ # assign samples with conditions and true effects
+ assign_in_order(
+ data.frame(
+ SampleIndex = 1:96,
+ Compound = factor(rep(conditions, 8), levels = conditions),
+ trueEffect = rnorm(96, mean = 10, sd = 1)
+ )
+ )
+
+
+
+
+
+
+
+
+
+Simulating a study in which randomization is not used +
+First we simulate a study in which randomization is not used. In this +context, it means that the treatment (controls and compounds in columns) +and the plate effect are correlated. The following plot visualizes the +layout of the plate, the true effect, the plate effect, and the +measurement as a sum of the true effect and the plate effect.
+
+# get observations with batch effect
+get_observations <- function(bc) {
+ bc$get_samples() |>
+ mutate(
+ plateEffect = 0.5 * sqrt((row - 4.5)^2 + (col - 6.5)^2),
+ measurement = trueEffect + plateEffect
+ )
+}
+
+
+
+
+| row | +col | +SampleIndex | +Compound | +trueEffect | +plateEffect | +measurement | +
|---|---|---|---|---|---|---|
| 1 | +1 | +1 | +DMSO | +9.530078 | +3.259601 | +12.78968 | +
| 1 | +2 | +2 | +Compound01 | +10.946764 | +2.850439 | +13.79720 | +
| 1 | +3 | +3 | +Compound02 | +10.111003 | +2.474874 | +12.58588 | +
| 1 | +4 | +4 | +Compound03 | +9.640621 | +2.150581 | +11.79120 | +
| 1 | +5 | +5 | +Compound04 | +9.466205 | +1.903943 | +11.37015 | +
| 1 | +6 | +6 | +Compound05 | +11.049547 | +1.767767 | +12.81731 | +
+cowplot::plot_grid(
+ plotlist = list(
+ plot_plate(dat,
+ plate = plate,
+ row = row, column = col, .color = Compound,
+ title = "Layout by treatment"
+ ),
+ plot_plate(dat,
+ plate = plate, row = row, column = col, .color = trueEffect,
+ title = "True effect"
+ ),
+ plot_plate(dat,
+ plate = plate, row = row, column = col, .color = plateEffect,
+ title = "Plate effect"
+ ),
+ plot_plate(dat,
+ plate = plate, row = row, column = col, .color = measurement,
+ title = "Measurement"
+ )
+ ), ncol = 2, nrow = 2
+)
When we perform an one-way ANOVA test with the true effect, the +F-test suggests that there are no significant differences between the +treatments (p>0.05).
+
+summary(aov(trueEffect ~ Compound, data = dat))
+#> Df Sum Sq Mean Sq F value Pr(>F)
+#> Compound 11 7.61 0.6917 0.657 0.774
+#> Residuals 84 88.43 1.0528However, if we consider the measurement, which sums the true effect +and the plate effect, the F-test suggests that there are significant +differences between the compounds (p<0.01).
+
+summary(aov(measurement ~ Compound, data = dat))
+#> Df Sum Sq Mean Sq F value Pr(>F)
+#> Compound 11 41.12 3.738 2.925 0.00258 **
+#> Residuals 84 107.34 1.278
+#> ---
+#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1To verify, we calculate Turkey’s honest significant differences using +true effect. As expected, no single compound shows significant +difference from the effect of DMSO (adjusted p-value>0.05)
+
+versusDMSO <- paste0(conditions[-1], "-", conditions[1])
+trueDiff <- TukeyHSD(aov(
+ trueEffect ~ Compound,
+ data = dat
+))$Compound
+trueDiff[versusDMSO, ]
+#> diff lwr upr p adj
+#> Compound01-DMSO -0.15660919 -1.881368 1.568150 1.0000000
+#> Compound02-DMSO 0.16041021 -1.564349 1.885169 1.0000000
+#> Compound03-DMSO -0.26495737 -1.989716 1.459802 0.9999957
+#> Compound04-DMSO 0.12799598 -1.596763 1.852755 1.0000000
+#> Compound05-DMSO 0.15392185 -1.570837 1.878681 1.0000000
+#> Compound06-DMSO -0.29447551 -2.019235 1.430284 0.9999872
+#> Compound07-DMSO 0.07293908 -1.651820 1.797698 1.0000000
+#> Compound08-DMSO 0.46903318 -1.255726 2.193792 0.9988016
+#> Compound09-DMSO 0.49798875 -1.226770 2.222748 0.9979416
+#> Compound10-DMSO -0.09048129 -1.815240 1.634278 1.0000000
+#> Compound11-DMSO -0.47509699 -2.199856 1.249662 0.9986527However, calculating the differences with measurements reveal that +Compound 6 would have a significant difference in viability from that of +DMSO (adjusted p<0.01).
+
+measureDiff <- TukeyHSD(aov(measurement ~ Compound,
+ data = dat
+))$Compound
+measureDiff[versusDMSO, ]
+#> diff lwr upr p adj
+#> Compound01-DMSO -0.6146694 -2.514910 1.2855708 0.994474856
+#> Compound02-DMSO -0.7383338 -2.638574 1.1619064 0.976093386
+#> Compound03-DMSO -1.5752825 -3.475523 0.3249577 0.204689940
+#> Compound04-DMSO -1.5418117 -3.442052 0.3584285 0.231362089
+#> Compound05-DMSO -1.7724524 -3.672693 0.1277878 0.091021614
+#> Compound06-DMSO -2.2208497 -4.121090 -0.3206095 0.008978303
+#> Compound07-DMSO -1.5968686 -3.497109 0.3033716 0.188684312
+#> Compound08-DMSO -0.8412919 -2.741532 1.0589483 0.939725524
+#> Compound09-DMSO -0.4007553 -2.300996 1.4994849 0.999893315
+#> Compound10-DMSO -0.5485415 -2.448782 1.3516987 0.997945430
+#> Compound11-DMSO -0.4750970 -2.375337 1.4251432 0.999450309We can also detect the difference visually with a Box-Whisker +plot.
+
+ggplot(
+ dat,
+ aes(x = Compound, y = measurement)
+) +
+ geom_boxplot() +
+ ylab("Measurement [w/o randomization]") +
+ theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 1))
Given that our simulation study assumed that no single compound +affects cell viability significantly differently from DMSO controls. So +the addition of plate effect causes one false discovery in this +simulation. It can be expected that the false-discovery rate may vary +depending on the relative strength and variability of the plate effect +with regard to the true effects. What matters most is the observation +that in the presence of plate effect, a lack of randomization, i.e. a +correlation of treatment with plate positions, may cause wrong +inferences.
+
+
+
+
+
+
+Randomization prevents plate effect from interfering with +inferences +
+Now we use the all but one assumptions made above, with the only +change that we shall randomize the layout of the samples. The +randomization will break the correlation between treatments and plate +effects.
+We use the builting function mk_plate_scoring_functions
+to define the scoring functions for the plate layout. We then use the
+optimize_design function to randomize the layout of the
+samples.
+set.seed(2307111)
+
+bc_rnd <- optimize_design(
+ bc,
+ scoring = mk_plate_scoring_functions(bc,
+ row = "row", column = "col",
+ group = "Compound"
+ )
+)We add plate effect to the randomized data and calculate the +measurement.
+ +
+
+
+
+
+| row | +col | +SampleIndex | +Compound | +trueEffect | +plateEffect | +measurement | +
|---|---|---|---|---|---|---|
| 1 | +1 | +1 | +DMSO | +9.530078 | +3.259601 | +12.78968 | +
| 1 | +2 | +24 | +Compound11 | +7.810705 | +2.850439 | +10.66114 | +
| 1 | +3 | +57 | +Compound08 | +8.419993 | +2.474874 | +10.89487 | +
| 1 | +4 | +4 | +Compound03 | +9.640621 | +2.150581 | +11.79120 | +
| 1 | +5 | +91 | +Compound06 | +10.952227 | +1.903943 | +12.85617 | +
| 1 | +6 | +94 | +Compound09 | +11.928334 | +1.767767 | +13.69610 | +
+cowplot::plot_grid(
+ plotlist = list(
+ plot_plate(dat_rnd,
+ plate = plate,
+ row = row, column = col, .color = Compound,
+ title = "Layout by treatment"
+ ),
+ plot_plate(dat_rnd,
+ plate = plate, row = row, column = col, .color = trueEffect,
+ title = "True effect"
+ ),
+ plot_plate(dat_rnd,
+ plate = plate, row = row, column = col, .color = plateEffect,
+ title = "Plate effect"
+ ),
+ plot_plate(dat_rnd,
+ plate = plate, row = row, column = col, .color = measurement,
+ title = "Measurement"
+ )
+ ), ncol = 2, nrow = 2
+)
When we apply the F-test, we detect no significant differences +between any compound and DMSO.
+
+randMeasureDiff <- TukeyHSD(aov(measurement ~ Compound,
+ data = dat_rnd
+))$Compound
+randMeasureDiff[versusDMSO, ]
+#> diff lwr upr p adj
+#> Compound01-DMSO -0.05930995 -2.268768 2.150148 1.0000000
+#> Compound02-DMSO 0.10429864 -2.105160 2.313757 1.0000000
+#> Compound03-DMSO -0.41812087 -2.627579 1.791337 0.9999637
+#> Compound04-DMSO 0.08581910 -2.123639 2.295277 1.0000000
+#> Compound05-DMSO 0.18471545 -2.024743 2.394174 1.0000000
+#> Compound06-DMSO -0.33999104 -2.549449 1.869467 0.9999956
+#> Compound07-DMSO 0.08156504 -2.127893 2.291023 1.0000000
+#> Compound08-DMSO 0.35967314 -1.849785 2.569132 0.9999922
+#> Compound09-DMSO 0.30525718 -1.904201 2.514716 0.9999986
+#> Compound10-DMSO -0.05702472 -2.266483 2.152434 1.0000000
+#> Compound11-DMSO -0.52416017 -2.733619 1.685298 0.9996658We can also use the boxplot as a visual help to inspect the +difference between the treatments, to confirm that randomization +prevents plate effect from affecting the statistical inference.
+
+ggplot(
+ dat_rnd,
+ aes(x = Compound, y = measurement)
+) +
+ geom_boxplot() +
+ ylab("Measurement [with randomization]") +
+ theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 1))
+
+ Discussions and conclusions +
+The simple case study discussed in this vignette is an application of +generative models, which means that assuming that we know the mechanism +by which the data is generated, we can simulate the data generation +process and use it for various purposes. In our cases, we simulated a +linear additive model of true effects of compounds and control on cell +viability and the plate effect induced by positions in a microtitre +plate. Using the model, we demonstrate that (1) plate effect can impact +statistical inference by introducing false positive (and in other case, +false negative) findings, and (2) a full randomization can guard +statistical inference by reducing the bias of the plate effect.
+While the case study is on the margin of being overly simple, we hope
+that it demonstrates the advantage of appropriate experiment design
+using tools like designit, as well as the necessity of
+statistical techniques such as randomization and blocking in drug
+discovery and development.