From 79b4789594a4a2654339c52861257a0ec4f11bcb Mon Sep 17 00:00:00 2001 From: Allen Xu Date: Tue, 20 Apr 2021 11:52:20 +0800 Subject: [PATCH] Update PandasUDF doc (#2089) * Update PandasUDF doc Update Pandas UDF doc with more details description Signed-off-by: Allen Xu * Resolve comments * More doc clean * doc clean and table reformat * doc clean * doc clean * Doc update * resolve comments * Resolve comments * Resolve comments * resolve comments * resolve comments * Resolve comments --- docs/additional-functionality/rapids-udfs.md | 140 +++++++++++++++---- docs/img/concurrentPythonWorker.PNG | Bin 0 -> 20840 bytes 2 files changed, 110 insertions(+), 30 deletions(-) create mode 100644 docs/img/concurrentPythonWorker.PNG diff --git a/docs/additional-functionality/rapids-udfs.md b/docs/additional-functionality/rapids-udfs.md index 2c659150114..4870bb049ba 100644 --- a/docs/additional-functionality/rapids-udfs.md +++ b/docs/additional-functionality/rapids-udfs.md @@ -134,31 +134,39 @@ implements a Hive simple UDF using [native code](../../udf-examples/src/main/cpp/src) to count words in strings -## GPU Scheduling For Pandas UDF +## GPU Support for Pandas UDF --- **NOTE** -The _GPU Scheduling for Pandas UDF_ is an experimental feature, and may change at any point it time. +The GPU support for Pandas UDF is an experimental feature, and may change at any point it time. --- -_GPU Scheduling for Pandas UDF_ is built on Apache Spark's [Pandas UDF(user defined -function)](https://spark.apache.org/docs/3.0.0/sql-pyspark-pandas-with-arrow.html#pandas-udfs-aka-vectorized-udfs), -and has two components: +GPU support for Pandas UDF is built on Apache Spark's [Pandas UDF(user defined +function)](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#pandas-udfs-a-k-a-vectorized-udfs), +and has two features: -- **Share GPU with JVM**: Let the Python process share JVM GPU. The Python process could run on the - same GPU with JVM. +- **GPU Assignment(Scheduling) in Python Process**: Let the Python process share the same GPU with +Spark executor JVM. Without this feature, in a non-isolated environment, some use cases with +Pandas UDF (an `independent` Python daemon process) can try to use GPUs other than the one we want it to +run on. For example, the user could launch a TensorFlow session inside Pandas UDF and the machine +contains 8 GPUs. Without this GPU sharing feature, TensorFlow will automatically use all 8 GPUs +which will conflict with existing Spark executor JVM processes. -- **Increase Speed**: Make the data transport faster between JVM process and Python process. +- **Increase Speed**: Speeds up data transfer between JVM process and Python process. -To enable _GPU Scheduling for Pandas UDF_, you need to configure your spark job with extra settings. +To enable GPU support for Pandas UDF, you need to configure your Spark job with extra settings. -1. Make sure GPU exclusive mode is disabled. Note that this will not work if you are using exclusive - mode to assign GPUs under spark. -2. Currently the python files are packed into the spark rapids plugin jar. +1. Make sure GPU `exclusive` mode is _disabled_. Note that this will not work if you are using +exclusive mode to assign GPUs under Spark. To disable exclusive mode, use + ``` + nvidia-smi -i 0 -c Default # Set GPU 0 to default mode, run as root. + ``` + +2. Currently the Python files are packed into the RAPIDS Accelerator jar. On Yarn, you need to add ```shell @@ -170,35 +178,107 @@ To enable _GPU Scheduling for Pandas UDF_, you need to configure your spark job On Standalone, you need to add ```shell ... - --conf spark.executorEnv.PYTHONPATH=rapids-4-spark_2.12-0.5.0.jar \ + --conf spark.executorEnv.PYTHONPATH=${SPARK_RAPIDS_PLUGIN_JAR} \ --py-files ${SPARK_RAPIDS_PLUGIN_JAR} ``` -3. Enable GPU Scheduling for Pandas UDF. +3. Enable GPU Assignment(Scheduling) for Pandas UDF. ```shell ... --conf spark.rapids.python.gpu.enabled=true \ - --conf spark.rapids.python.memory.gpu.pooling.enabled=false \ - --conf spark.rapids.sql.exec.ArrowEvalPythonExec=true \ - --conf spark.rapids.sql.exec.MapInPandasExec=true \ - --conf spark.rapids.sql.exec.FlatMapGroupsInPandasExec=true \ - --conf spark.rapids.sql.exec.AggregateInPandasExec=true \ - --conf spark.rapids.sql.exec.FlatMapCoGroupsInPandasExec=true \ - --conf spark.rapids.sql.exec.WindowInPandasExec=true ``` -Please note the data transfer acceleration only supports scalar UDF and Scalar iterator UDF currently. -You could choose the exec you need to enable. +Please note: every type of Pandas UDF on Spark is run by a specific Spark execution plan. RAPIDS +Accelerator has a 1-1 mapping support for each of them. Not all Pandas UDF types are data-transfer +accelerated at present: + + | Spark Execution Plan|Data Transfer Accelerated|Use Case| + |----------------------|----------|--------| + |ArrowEvalPythonExec|yes|[Series to Series](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#series-to-series), [Iterator of Series to Iterator of Series](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#iterator-of-series-to-iterator-of-series) and [Iterator of Multiple Series to Iterator of Series](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#iterator-of-multiple-series-to-iterator-of-series)| + |MapInPandasExec|yes|[Map](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#map)| + | WindowInPandasExec|yes|[Window](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#series-to-scalar)| + | FlatMapGroupsInPandasExec|no|[Grouped Map](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#grouped-map)| + | AggregateInPandasExec|no|[Aggregate](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#series-to-scalar)| + |FlatMapCoGroupsInPandasExec|no|[Co-grouped Map](https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#co-grouped-map)| + ### Other Configuration -Following configuration settings are also for _GPU Scheduling for Pandas UDF_ -``` -spark.rapids.python.concurrentPythonWorkers -spark.rapids.python.memory.gpu.allocFraction -spark.rapids.python.memory.gpu.maxAllocFraction -``` +The following configuration settings are also relevant for GPU scheduling for Pandas UDF. + +1. Memory efficiency + + ```shell + --conf spark.rapids.python.memory.gpu.pooling.enabled=false \ + --conf spark.rapids.python.memory.gpu.allocFraction=0.1 \ + --conf spark.rapids.python.memory.gpu.maxAllocFraction= 0.2 \ + ``` + Similar to the [RMM pooling for JVM](../tuning-guide.md#pooled-memory) settings like + `spark.rapids.memory.gpu.allocFraction` and `spark.rapids.memory.gpu.maxAllocFraction` except + these specify the GPU pool size for the _Python processes_. Half of the GPU _available_ memory + will be used by default if it is not specified. + + +2. Limit of concurrent Python processes + + ```shell + --conf spark.rapids.python.concurrentPythonWorkers=2 \ + ``` + This parameter limits the total concurrent running _Python processes_ for a Spark executor. + It defaults to 0 which means no limit. Note that for certain cases, setting + this value too small _may result in a hang for your Spark job_ because a task may contain + multiple Pandas UDF(`MapInPandas`) instances which result in multiple Python processes. + Each process will try to acquire the Python GPU process semaphore. This may result in a + deadlock situation because a Spark job will not proceed until all its tasks are finished. + + For example, in a specific Spark Stage that contains 3 Pandas UDFs, 2 Spark tasks are running + and each task launches 3 Python processes while we set this + `spark.rapids.python.concurrentPythonWorkers` to 4. + + ```python + df_1 = df_0.mapInPandas(udf_1, schema_1) + df_2 = df_1.mapInPandas(udf_2, schema_2) + df_3 = df_2.mapInPandas(udf_3, schema_3) + df_3.explain(True) + ``` + The RAPIDS Accelerator query explain: + ``` + ... + *Exec could partially run on GPU + *Exec could partially run on GPU + *Exec could partially run on GPU + ... + ``` + + ![Python concurrent worker](../img/concurrentPythonWorker.PNG) + + In this case, each Pandas UDF will launch a Python process. At this moment two Python processes + in each task(in light green) acquired their semaphore but neither of them are able to proceed + because both of them are waiting for their third semaphore to start the task. + + Another example is to use `ArrowEvalPythonExec`, with the following code: + + ```python + import pyspark.sql.functions as F + ... + df = df.withColumn("c_1",udf_1(F.col("a"), F.col("b"))) + df = df.withColumn('c_2', F.hash(F.col('c_1'))) + df = df.withColumn("c_3",udf_2(F.col("c_2"))) + ... + ``` + The physical plan: + ``` + +- GpuArrowEvalPython + +- ... + +- ... + +- GpuArrowEvalPython + ``` + This means each Spark task will trigger 2 Python processes. In this case, if we set + `spark.rapids.python.concurrentPythonWorkers=2`, it will also probably result in a hang as we + allow 2 tasks running and each of them spawns 2 Python processes. Let's say Task_1_Process_1 and + Task_2_Process_1 acquired the semaphore, but neither of them are going to proceed becasue both + of them are waiting for their second semaphore. To find details on the above Python configuration settings, please see the [RAPIDS Accelerator for -Apache Spark Configuration Guide](../configs.md). +Apache Spark Configuration Guide](../configs.md). Search 'pandas' for a quick navigation jump. diff --git a/docs/img/concurrentPythonWorker.PNG b/docs/img/concurrentPythonWorker.PNG new file mode 100644 index 0000000000000000000000000000000000000000..6833c4a143170eec62aec2c4d672c2911aab42f4 GIT binary patch literal 20840 zcmeIac|4Tu-#=VeD=nhr5+Q5I5+S=1+0_VHhQ!#7j4gYMQnK&6Le@bBGiGe1vdtJ| zYcQrT)q16H9^-O1jToi5ItN~e-|_aA2|Kfvv9b+YbZVFe=flL3Qz%NOS=SZx`k$~-taX!q_HMj9I7CAQ&`>x%=J$E=xL4*W5GXA?9~^8gp&ldt}m3o zDW=atQ!l(3E0QF7E5%tg)S0zG=z;t%dq z@&`b=6QcSdfd^G|X)Tu=MS%t$k3kN2^Je9NxVbNSA1<%)cz+z|5QleRYTk8jtDG?j z%)OoORx72t$7S`6IjEY;?*3)^nugdsW)8}2 z*ITH@^ZD(P-%)YXpDk*Ldbm6@su}51xH{<=|3oWCj;x#=eTLq;;t}<6m9%2-Q5ZOI z%Iqf_%R1UZI3)wo>%U#Oh&x*%bV-BA-_xxkE6KSkR6gUPs_Kju>M#(1jmCj%y!tK0 zjBx6UFBdZ#4OD{Br=JUd+b^A5&HvfBg+KQpC~~ehFhpww+YA2SAePP+*A#pGg=K@( zZXEP@{;WJmJO2VFhfsJvT)H+Yj<8}bt$yY4z@=TlP$c#CoZN+IG)Od zirDf)=tm@K>D|k5J>;{l+={Lk&H20nng26LK`wisM}b|l6Tegxc7Fb^U=7kXiEAP| z$JXGvXA^IRjVCWL#Iz*Cx`}Z~5mBX6Rh8O2A)=b-xbA@j*Gt6mKI3jaJHF}Fgyx{D z9{mTbPw^`KaasRK$tyuE5nX$bkj-#?;Zq?p5o#>2YU?cPMS`MZ*^jR`Kc}}|uW4q=T5?cv+&~nI}rJqaM5Tw>P*U} z4qoRgZC$WeD2RW@+6$2RW~N0g{!xg+MKvO$iC;;Hnb?X#X;`N{GxMyIC-lKLcwt$a z6z9dTM#5Pm&j8BGVH}6nql5SJ4@O7sSC=LtG+nbdvzYc%=RftaZhQj!136@gH!3XV_)UpL6Dm>WWxp zQM^F3bsH8x7m28x=#FLB()?8vt(aaJyEU;|^oW%kp~kt$5swY&0I4J>UfK=BwK+Ff z<|R!;KSS1bXI&;86=(bTf_>;K2aD9u_Q&$pF7kbzOX@tIXM6dIiVZ5N z%=VI@g795PLH;6yW*+MRaw8k+t+VA@$|F9%=4YyolFl}Uh8-+VNkw$8cZ-P)#yGeI zDd3w@Hv+Xrp&I@j{$M}9EEnigf=U3|S2cJ-JeK7?OGK0VsCb;XSafnkVr)vxqyx;R zUOwj(;hAXlwS?JyyF>r$s|rSxBiU-rx)dUHHJN_0$po{QSIJp86(-8KI2V}WnwOJa|VUL*1I+`IX@Moa_Q(QQ}Uv|+d;X}IBO zT}`fX!Ectx#JL1Ca^P+NJ12E{qo_O)oZf#^gL4&L*}|C3kgj`Az)}P1ZRlsFun0wo zapbvIm#!+L7xfla56TBD7l>#gQ{1o%6vK}-Au@Dn9^sx#kLNiix+2d;Rx-z}YJ)R4 zHS4FpM~rje<^$S=?gj9gVB(1!xQCDSZYZPQKecXZF>zTpfS@d{BZ%;AsU@l_+lJXv zx^wO6J;qMDlWI?gy0ZA7N|3ibnd8|9V(wt1t#ie58^A>2p|zwluj0uYH(Ud957Krp zBGM=3UdF*ZPFc$-?QV5ToVwtIFeMDVHUEq=SaSY(%0fBsk>O#A1X|BSrQp-*dy>4b zE#*asFeGcNLLb>}<&RVrL`k?c@zVa>;Y1irUDzWNZ+h!$$3ZLmWR8lvgEhS4T|(Xw zcqTn&bR;phPazTsuP`IGtCvOs#rHQoC^9%a?1)diwa|9S%kq?U#;?vmKMRF3@j#Lm zO3N=sht8c3T>ah`D%=FZE2^%Hq>SFxdG&z0dpV>m=vDEKF4pRuMh13Gf}FrPMvoB~JaWGQ1x+w(bK#M1ldCP1;4d;o@qYP=y|$pbET_|W-W_h=1@?wjw+opV7T$EI zpDGNA=~|Cr@72_RL-@LFZjY7^8x`0!Ho*l7T3gq>`df3vw&NdwF`3IuEXDlNL3stP zNK^|pv~1PAGITGUsom;m9X5R4`~YQF<_61$EVt3ejksK!E^Fa+hnlv%>B^o#Clz~# zSrS?A-OhGi<@E*6u*zlZdJ*|t>wS*=-HDfH=-R>JHPep(%vXw>fGx4AWuWYfo*9XjQ zl6J9neS^`eZ1DU|9)7yxS;eMK7eVVr+{z zXzXr)xoXiHu9HcR?*X@dvg*2&v9XlN6v;~yhLK#5$S7tUqh^3tF`S)+M~Gx=xTrcw z6-Xk@j#@FF45Vp00%V->3UB$`#}Xqji2hgZN!$Z2wXX6{jdf`|hWgz;KIbc`+aM_h z1W{b>ZNITmng9~a!c?=@HHMxCIcHx{ljWkGu?dGqgm=VfTMLEx33tRzV6ks==Fno` zKYDuK+KE7Q&O;V|DAhg|RnG!oz?jr2-j`)bb(k6aEJC^gpwRZ8`#d7Md4a9}ndBIp#@0nVoPP>EKhvPdT8VWUgjhE5< z;#EE8fhAbOp0|{g;&^W%zIlQL z^Ob7rcxG^W-5LG?@fkkqn4<=ND3qFjT8n|Y_)N^B2S28y<_lfXb|(|nzLc3nY6NRe zt)JS-%c<|Mo`sq@Rn%QFpLb$UaYt%i%hpSqyU?K?yqnzr)gX}3j(yyp=y5LT_NV8& zN#0JEE%Ogr#w%GMUpQIZp3p%fg?mfro1F9f!o|hBu}+CdP^2u1GS@Q17Tt7TCtnxN zu7VT*;ZjB2_R}>IDBAqp_Bz)-M+e9B@LE~bnRidqnnkZJPO1t zl({0&NfVF6iOiHU?1Vlzq10$*t?RL>mlr4+MqslMG8xVI+T3)ab@fWP)22<3WdjvE zQ3vX5Jr(}!uaol=u|Zu#orQd%CpPeyJb1f7uS?jEw$Lus7ivr#ne;2{P&L|CAP6*S zW%Ix5KtHM8d^2j=Ki_eG1K|~-%+97ais*%O499v;&NIt_v$Q$106fxvFm^QD)?HSBGrhY5C#d+L)syWZ-lhA^MMw`AcB zJdnLKmlX88Bmay?$He99K7CE=9@xT-dtVJDLqz+Q!OxFU_;Q>1>NjVP{u}}? z0SxInzjWNVC0a#Yy41D>A^<}0G88uX_Q~Mldj5RdopPej)L_ua9G%zfJ-|-w2!UU! zH=Fxln9uBUX-aN(Dok-vz|)z3S=E!x&@eEE4Bz%zk^KAm9q6hdHAC z$f3N-`voGB-I|9SH2+9S{abi;fQ0p**l99y7&wNmny3zK#{CI*Peg?6<8}7BS1s z+V(v_LePH_IPOR)(iAp{nq3IiB6>E+#;)j~$p|rs&_Hio*SdoT+F9yt2KGu=mjKR9 z*|&T~1LoT(gmOJ)y$wJl8XA_J{zr(G!i3AYgG{WJworwocT#%-P@G5P$Q!e?rP91c zdWCIogst+Pimo&1@$AAI;MSE^p@|3_U+l9=JhIBFTzN7%3iFjBv)I6xEf8H~T-a&+ z#x_(v347#-#ga(2MnuFrUN9;a}lR`}~TsKdfSr~iq365^rC>zvz(%z98 z{^B#eWgYLRQGToNB;mxOKiH#T;&0z&EES~-a3XnUA!~%=w^|)teZNdaIWbyuF?*3B zjZ6EGkmDT$bCc;Ble2&8al*6l;Jk<>kJN0)SOA4a_CwvB)iTF&&SXecU8iDe6Cbvv zi1@AP`qZ_A%~yF0+}w`9f8z(OEug&)~(&|j_qOA6N({6&yV!DvjDr7 zR_rbUwKjAKM3)FR`1R>K%x%X(x~3nO=faDy&tAH>qb#9rsG4P^k?s>xQa@S}BoqsW z-^aR_%)Ff@4J7GO_*f1_ZPsmIpig%;C~jS8%a2jUrOu6PWc-b%Qrj2B9UNM6GJ)F4 z;%!x4=W}VTgdt!~?d?%&X=aDTPlJ3ZzRbZz&sMwK*Y|g=m3PH81Ur$>>6#M?{`OfN z9TJw%I@Av=%I(v0KQG55>+2UK5t9KEpO|HpRI^U)A zzDFr7)>*^jl7)(9(Qhe{)`=w+iKeq|&n7N^%C}#fYCA7;)#aVnv=2A#Tae8%?{Dkz z5~s&Aa3rkjEQngAB1`1Sg4*xf2II%C-^nO3U9GqmTbSP+&w~>wR&S3r*Pimp4O?uq zyT;$FQfw8L7njU+O~@t(zF1Qp5^|M%j`$|r6~?D{p#z;9MJh8lG-tO<>AUCAZZ4c2 znX#rte*X2a6^ytVLJq+V^IBQ+@R8S&`V(DR`EPg&;JC!*8)Z^rrhPuu9QkeAe#d0z zMgt{V^@`U>0?^bas>0rHqDalb#}!v-mKISOIwf-xUZfdnju^0HwCZ(gX=&+uq6cGL zWV?yFHO~bDZOgtrNIBYRsM_SwD>EgLxZON361)Iwmk+~CK5h*hgkY6`?TN{`2l6N;bvF}t(L zO|i{T4vb#wEZ`F2^c24|-+iJ=0AWR8JDGU0PHvlt_{WkB;;j#wUjBQ-8e?ZxMH#A4 z^H(+p3@o3>dlpGc=(Rck;N2Le?(}{B0Qg8}RyC zMa}Z2l^Z{oN<$%{oOizTyfM0v;2ux8oM>9!T3qCBxb;%ghv~N&LG3u^G{_#|)<^8V zmXJ3I`cfHU8Q3Chg!1u3w5zaOD)?afNg`a}sT5xdjoNe9axo4YSA1Yb5ctyg*H}@y z<_7%Cx?MBq_x!>vt9@$q;cZa1x}x`WjZEJo7XB+9nVN{0`EdFg=^h0%yJRVGs1FGreknNqcx2V7P~C-$k%+7}JDDMc8FjfnoohL6f+uL|7D-_sllhCb zEfrm!y|-ud{B1?}OX5=Rrb9;6rT9wo;R=c~qb|V4yJ7Az{YBo3gyt{XfwI% z(w&o7>;2WM-;k|a)5sm{^c${AEkjTG?UJ=8`YT>57ig1QCCnFD8!*!EGqgx^y6;QG zx?#}=ERFn>pTw~+T^-bvuzoF2GU5s<^l5e^NSWD|4c8CEGV`=!MtN!6hin{M8xh;# z!3NEA&qIsnM=0B+4G*5qdiX61uO+qjP>cCH0D`%y_w%P1VFtP9>QPZrMKxA_4Q(fP z5km{QcpwI#J;V)ORMOOGd6-o$z}=_zmegdz5%cCr!!#*_;N7dWb5H)dnojcD3a*k| zgO}V7Ip^bB#JhTE-X9U*x-d6Cw?6xH>GJ{AT@Vy-Hm){~D?Tk+nW=@5`$A~0!?09* z02C~KY8SUGAK4*U{1-UCaDTYn^SnlNq5O|M_;$@O5VQEYvOk28cf4;**oS!R@uiPD z=o7d3Y}&qKNY5J-ltz;$h^@Yxqds_kS+)lNU|FK_cx@@ihr2jYv0a>F5wX9zTSw#; zsX+FR$Q6}IqptMSAkMO-Rifwo;j4r=C!TwBn3%> zjI8i^Z1}MA$&C*f)q6J7VnaLQcin*5w2c*WMJdCCteMPTKzSdHif8$F_W=aAtcp1o zhUv*P0Z%PnWHjWs#V6-I;m(r4V9ml^m3KKS<_O?@{p_`*%o-c2D=q`Y$%qhK{)}_7UPJ zfnJJeDEKGUSuaFAPu3l$Qf?8OmvA;_+SnD(?uHlK0HsyA_dQabBYoee=bb4qCvcgv zS4prz54lqX1?klQi!n_BU>M)@tWQ7B(L^zy)4l2K%sI@X4zY#LNyC!;PlBf~@B#D4 zp_v^vOCnLGmV*hAd?D<7>tC&>#KwrTTwH&)wVH)TUO=CoCNr6F=o-1V%O(83JTh zXTjXdzsM{g@!KI9*NqZx=Hz?4LeyMfVr{S9E)=S$D>Tw2l=&hY8Dns^^NTgzcA4;i z^iC-ri7#iqwl8E?SAUacTHYx8ljOiQlxR5Gx!yZ!`LdTV&{bl|y4S2mLYficnQD&> zxV%vgCRb}Ma*e;ZDWYKskpugsbe(=i$Fq6@x|_^g}W?PlV&%p(%w&| z4FzZR9czSbcl{OocRaTEBx~oQ5SJG&)L34t<_wbNf#gxQ%Hr4-**5n+Q(!0 zfyrNh7w8WIzgFxqugxb;O#>*CWWt?F+({A)WAAL!x8%+x;d!{|&JNOje%7iVK)TxV z@~5ols565*ug-VGo{)B}fG`lDiN9p*nus|Ae|bY9(cUD8Fn2DqGpW=f8LJ^TDTvL7 zk8aZMe$C4KXh(Isk|9b05j5ZOl{_8Xu0!*v=GbJ5W(}gbzKtOYj}h%`2d$-RDq27^9%=z z37-*}40s_*&H$(#GiilOOy1t>431qG80!F)Ma7j)ia3meg_&u57mPqkVl|Em;rb4Y z{D4)qUme=`KG%e>XY@CW8p1H<4(E+`wG0x2?#z%|M(KAWNQo7lqOBB2FH^YdRVn<; z(JXAOnG}nA-fL(UjVgkxfA^`(g~wSn3eDre%*$aN{CjKH$}VDPUV9!AzS6J|rUO@p zesonDecZa`s)=o50;&rDg_SRnxL(%*V=<06W~6JF%cg8Yf;bGksPAkZ+<*sV8lA+&DD*sgEJgSR&$)tE2#j59VJ;OeEC)xql2LD# z_ zEEkquOf+D*uZewDHP0O1!xL*3c?T&_KeX4CG})y4Tw`__F_*d1gNU~ z5TE*R|8 zn4O#qNC#baIXEGhX(hXr>hGeRjVJV(&e{|`#Ci>-_~@U ze`m6$jEg6d!Za;un4saS@Hp={KYAQYiYKRdR>qc@ zHczIuEK7T~LZ`^tGMDlyEU@Y*tsuP63ar`dN10F68)r(o;PM;*7D_qvAm9vP*Ppx= zN&y8KYdPe+Jg2fu7sM;k#gMK^iZzsL7P`~hgmyb0IX5a<#;sR?40Mm5wYSg~PHkNY^Hd72!~|{B@S^+Zh6_ED)|Gi~B@{($ z-Z?vsjBypaQgG13{*u0(Q|z+X&ndycrFD~oc(n8ZmV|K}L9#BD^LD^iG;WJwix^|C z*4oiWGIZs!Oh$D?$6r}Y(gKgSH9KjGG>Ezn06VPvCvEj9(bbE4a~SA2UZXj`2?d$f z^E5ePktU%2p%-$F%h3ZUwNx^l1DAb3$V7xyjNB%_KO%?^Jogc8td70an|SMvsOCU| z*|B;396_HTMK+Jqzutsa9<-6CBa7XSDM@wpWSy@>-&oY8tWC?nV$(%dtQ|d@nzXiD zM^<{DikIfL3F-XJ_}j&mo!#z9kwfVu2P5ObP{9oji~NcxI$K7T)3+0H5E4c$$gpC2 zyS{)kD*>(5Ki8KCrmf8EUd*WDd_O<6>8$Jtf?gE$i*sECm-tew7 zj(vzICquUJk<~7TC~O@3<&`Dasp@w5@XDy$4f*g&O% z_RS8L)$upmVn9Bwo~bEWXzX$5=G~-;Y^z@DXc;Ntu{QgTWz&`-pB+9-xn5+A%7@wi zjSmCcR#i8qYYJ=TAzb^8J}!K~MCk)mPy$gK|KXQdZTaZvTInB0R%dJdp>Pfwm(1eGTEB6+ zZixds+sgb0K**I-`;g8q>iw95vN_k9@#CGuG*j?8(Yp-{&8CfBaPeP_?Ak>P``7jn zyRsW{7P3ytp9YDrf8F87+DYiG>BZyr^88xit{o>>*Z>~w<`|H~5+0d3T3C+1l#Z9f}iatgs z{Z^K<=rU>@HdE=$&c7?#?+2h4AP-QuXR+$Ay18=AYqR;-T<=?(LfQpGAw~gp|Nfbc zM)GK9`H!lN>3KuA!+|=g+yI|m%0vcmqVP=-MUuhes4cuhW@~e#W~))pQ3`e07R7LG zOx+Y|e)DRCHu{vG)0o3SyJ_l$i7nHwUvY*Kl~CHKJgKAJ$;l>~8w>FwIoq+ur9`b)o?9J{C{k23^ZgAs;|`c`*3 zNGGvq34*}ho*Mi*`%9x#FZ{Spj#E4w z9)4jJzO@>1$*A6OCgZ%RX<2iZrcI2TZle|S`Un=;-&}B7tk6FVAKKWM03Kkp_QMM6 z`Nc#Ja`_}n_()EIYh~IlQUz1YGD4`0_){7j$CC|bGRrW35E&y`pv#U* zls?eQ7CHdH7cay`ew(v}Xe||u?i(2riO&BBU!?SnTGzR7!@mY^Kg$WX^8P`mL{b?5 zU<|iyzZQ#gWYW=sJU<(lJHq3S*R;a9%8gZ3efv^bSg$$=3N3^l4d#L&Rl14nI|T8r z#?H!038aIfd`YoW99IEKz`MdedY2xviI<9i8jlYs$nAcTd)U{dWzBaNzx-06?h;)m zyd|B@l$NL^_MI8>U)I(Ad&OQFh*zgElyUn+-(-1;7D>idxK@uB-?+iGp2t6>d;K@{ zpVv7+eaY4AUC8k{NSa?N?WRGbtax_M!O?~5s?BV(oH+Kgy9y)@uXAYL&)(OpeG8i` z#hesZ3trt!JkdCY@FSTLQsNWKU)wlMYwraoV&f4yzO*)>jV{vPa9Zvxf4u5&*_!mq z_fDVFdy5N2g%T{wCuPhI-^k*8%UB6>j#IqgxzN@(qAjiHqo)Dw8P1%9bqk5RT+i<5 z&YDo57`Nca^jFs_C*-grTE?I$)zAYZ)xbT@Q)mZz|;-r=u2kYbAj(f_8@^7#CZoJTm( zJ{S+>v1`9uZ~5_WoQF#I%_4XClX>&w`K~A5iq~(-0vd_~djhaQgMLJ8^muYh9Pyi@ zDxA-Gyn8gkWPm<9V~^^H**E1!MKSsjb|?ucDd%H4O)i~2ti}pF%0jxZkZ?6 z|2de13xGJgeh#qdHo>!u_A7h7@o0=c4MJ?aezA2|ERdOwTAu=Rywz$CKFhZyeBEO= zRu5*2We;b2c8!wtW;_HBL>c$UCeI8Z+k%%jh{ZLR2*flD#Cxg0!dx}=h;C(r?D~7J zQ;QR7uNsNRu^ZXKwK)Vg9T(J$ z>$AKDxlIUN%hke0gjct%vaJqN9`W2%)uoSq+d5b9P84E&uPPB@>isrGtfZ50sPX7^Oclftx&UHQ;Kbpm25jVHh4H0*4 zR5y2kDe2(NqGbWN$*GY-w?X_ER3Fwj22u?UGl4j0kJ!kT!C06hB7Hyfhq7MyO|=>M zEqEQ!e}D1$*L=nYfgz+uWz)A78k;|&zMB>vZzQCMzp-s7k4ZzwN4z~*7joq1B4YNb zu!20Bk);*wk30)O+^%^ls_0wU&STWP+HjPVGS7;QY@te1y04DsJk-Wd2ITNM(iHT( z?ZxQOH8+~$%iLah|0dA$*g$Ml6bSc!`>A>SUlZyPS;afSIiF)EyKkKHk$y2Smev4B zWyACfF~2hz)v=KX@ANrkAN7~zluv7yCaU7(93)JJ@FQKsN{D;dnw+0e%y)P0c?ZcIfIvhbnD zWXgI0o;GS%si&m=*)4OfBrcitPwWH>Ls@^tdwTQy%LnvdN~A~I6ZNdIHM7?(>nUF= z#WJ~#m;1TjKsQ2!j2?7*dc2OA;V{*O%DI`HQgW4q&14ia`%wuA0FWFmex8`{#V^gH zhr?@)nsau09-pG_dpqtAAcYKJ9E3g*jZWQoIcAh zpQqd!*cnSUgLiPxhOMVgGIx4|64*wwStR9-o_-zP_U|YQz>jkQP>ufjsRkF_UT!5_ z@L}Ee$e*(vs4I3&T7~~NOY;76{Kijurim($>^F37W-&C6GT$Dz5!)bOVPhX_`TS08 zf+uHue@r}bKQZR8AG0eFTz~by;?1E7)RyV~wLOO_A?s)V=MJ46d70urfeJe!u$$q! zS-z7gTK+OSx>=AMxEZ``LR9jkm?xGd?ZTeca@_#30)&%#bDcFaBh z8!bG4$?^Vg!xDRXkFPvR2{|)l%2{%EhL%jmdWF@oq8P1VG37Wj@%1v@?-6|^$lZF0 z0Lq*!%`moNlg@VT)w~7R8;m?n?5}I!k@M4%v7&XC$i~J#miQ949;ky+dRA)1sHFh= z;8&M_XzBl&n`cjE0~n?L16c7ZB5yS<*v?ftYUV45u;&!@5mRDmqzRxR!@3Q(E@mmG zsry}*-RO5R{9mIh9{Ma#3|(Y?D|#b|*ozZEVkuSw$`PM=G)lU_QYb!M4LIHIsp zkTuXtuXxyU=>?~Jv_wMq)RSXC|I&Z3Ou&Bux>+T0l)^_Rcn$=-8JAHXqgZYm@ zh9aQSH2&1E$LC^WH|2T)QxBxvbP2^9)IY@kQn-IHo}{uP4u10TMdcDmcjNLDpFGvN z1Lhg2JKmTF7oR9Ky)qrnup{HT#{jFTu=Z2%v3rR=PM|7fC;}hqe98BL!moQC8vHuA zS5?3Mrg8mA5?XNRkgl|ReFbIeiv*3UjJ2s>{LX*RAZ7b;lpp}?>CeIR~ zv@?}atn4uCvle((%Hx*0p3>)=?Yj!*Jw@BKI$SflGl~@&NmwWKwlwy~RJM3+dJ4kc z>*3jpMPn~9rNq6pe|FMc2s;V;){MF9x;()GyHGdz^zZF_03+(Pj7H|QkWHEfvSl^= zd9RoQ2&$$Q`C{YwT^faq!LZJ|9zkO?irqxQccOX#<>PTl;_^HMHoFmmML%3 z?<#$N(k)}I1l1y)mrK9xEE>8!XEn0n75-`5yv9N8jfW-*aoIc~hzEY{2Y=H2dW*i%71 z5+)QN7qZNvDN~HhwdKNSSm`U|1!X_yUT-ih39x_EJ3BY#lTjEeRpL-Ntsdl}2=WMu zM@nKe#v#1#OF5K?qJ(6wA%vN{-F)Q!jjZq;^r3=<$hhr4Y6@?N&>5ARjgqEA`!Y^g zn~W$()FS(KBh#eU{D}r#GFWTj1E}uCO^&pX4~RXS4;L#%Qkg-Zove*BnR-@4V1-4`9rKbb~u1}9yECTjU|E27h{-UN|jyIo%@iX34&Q$Mi zjpcZmA`Wc~ju{UL(3)AO12pw=L#y}$>6@dH95$uQ{L(^_+P?$s;2gm_;TH__3q&+L za|N{)3A#y3K_P4x=@Nn38#YIh50ql-{<5pjxXd2dtf;uUFijI zH${Cge^GiQ&N&R-g*^>JmVifJFubmA6)~DAp%t4#j*N^rec+c4^A686$z7$d?1}^Z z;Q?@m9e*mDMz0oAoQhUvaNbnna9z0aR-v!-1)fQ|V4{2+{9lUJL;K1RgZ=G=P2S-o=LV%?YSjL=MHOFN|$ovc5ztQ zi~g2(I1_A=;Eu#F*&17cp#+RAuGWs7eWw^LM8t+t*Cz{mkZEsU_uJn`akbhzQSILh zat{3W<~067kP~pcixI&-PcaO4Gcg|WVFU_7d)v8BWRCDIdl<8f5>EC~)>$)-J^+I)CH3=H zX6O(4wsB^aaEW1JA~S{z2*%N?gp-B+s)wz(%|$)AJYy}z(C4^`)xW2f)Xo!W_$7=m zQ_x8cT&Fw#IJo|Dopn6CuUeKcmT@Ak>t4~lSQnGGM5`ZvH_nv}8Efbgz)f?neObR= zC-&`W)&N@z;EJq^X?PVSroGmF8gz~0;D^x$rSx~8kjezq3~MYu|WhIUO7>>STBg}rqT1EI-GaH zAr3&9`e4AccC_PzvB8(Sv>-j9_m*jMse$8>@OrSp;3Y#lMFYKePFRh!iYRCp^F&d{ zV6(deXJ|F+4QIF3p}_vz4PQ_xrSWN|&TiOl{mJ$umx7tH%z>PZHZ#zDA=wfI?8#_o zKcNcagM#C4r{{>g5<7q2s2MPKI1tw0^+Q$}g-1@%?||c-EH0S}vE~GoAr*aBFjlVr z#7ZZbkQm|H<;pravQI-Gn5@>M7lBW&3!dzHgEE%SN}8ty}-FPk6~sBL znw!Vj)URnR?9KUIF?X`7c{Tq{VMQDGDJoq^sqz>%n*2*HSxQ$E(^D+xyh|ohs{F}^ z{<^C8yP=K?Yn&j_B$BWky_uXDpB_&NXD<_B69sf8fJJMi^k10j0Mvq3+%b*)pL}(@ zaPps9>!{(K7I8C7W=sDPEwiAd1~!`hEThdZQ$a!l{1D%>aTh4Y;^Xen*nF9`KQI2o zSh5g3+qd@e6D`16zK;KA zr*{CSXu#}mhn=tG-Ck*4pUY$l!yeoA|8VuqD{i3oTkVSHuiCaqis%}X38K29*5oP9&U@W%EwzkTh_44zm$*GI5BbpG@dTfgB?^IFO`0C zwz(ZVsjVbfALsZ`n);6zirQ7{q00A@R3~RWZeOD6B@4FrPgizx%_zu!2 zDlkZ&7o8{?q}B6V`k1|WVTjgsG5OHYmf68wX=mdAZK-gAa1?8KR z;fgsGYYPv8=}Q$NVrN(_Y7Api1Ql(nL{PnD-zS`$1?_bdj64QzUi<`W}l&ZjhmcnmT^jU8-0iA*MOJjj37u9d8wokvyBx zF~yVPO=usyw69NVh2ouawZ+=%)iz&AnR}9J6AOU}edgtm;`0>>k6t``+wpyoUZreYX^cX_@0e-|e1;ESqJ zop0lFC05Fwkan5p+%+z?u z7313TwkdsWQCfw)RDeG*#9E^F7+?Q z%v#wR&>*>`b3PCB3d}(Ew8yHu$8mg*t92*foTMoZ+S%kEu^CozU_k*F{|-)0o_sy2 zZk`-I5xWVAM9h~rv3+XBVWi$Go?J7$GqF}(GLhQqDv7>lIDf`4twz*_2hug%C$^kd zS@={`H#Y0ulvpUeWTh2xns_W!xc8pXc{OKaNX8}f)gK>%#oyEe2XuT9i}Bz+GgT4y zMGqN|C>wNx%=~Eh2oOxW>ha=d{9Px|R!Iw)GuJyVlCC8bTLT#5Ss8#aVhhqNr4iB* zu%w58;R*F>E^Z4|BO4~1Y37+%IC+JuL9w>yi_pYxO2xie?A&G%=0UP_Y+;i<(%n~| zy>Bb?{kG5ao+5k48xs2++HorTX*oQr2Oj>?Fl{UNpP{5pO_}@(t?DD*;7IvhmPS# z-%YbS)EZQGzFG9_0FD~l)tk)im{jB!%&Dhi*ts=j>tl<)9o5t$#%T39f=25Poi}zm zqH4fhIV~OL?z)eMu_LrO$QTN{cd5WxtqnHx$4%e(ox?kv_214_=V%cDRxndVs*IbT zr#)#M^(ey{UE;GDqNvrM08d9YK4XB*>DnW5O zXk#Tlof$ZfgPV#!E&v~BFdRVDSlUvLii)soC~hKcVslGC{o1Fj>oSXeF^g0q4qxq6 z%c3)$^Z+v17$%E%?=Uk;*S)@XZO7OEGQu5~lUOs-iVwHmwAG@x(bqt=;_TX-l&-JN zB&KY%JXUNB-(GYDc%v6AexVyvngQ>Mn0JnNOP~`D5K`Ry?4uTaP}Q7e?Ob~gts7Nr z!V0FJ=za>E38$a1W6Od6v*SFxSKsdvg;a}AvCXYXbqnv5Wa!r)R6_AOpkc!v22Pr& zRsxf{R7kX;+=>9?kSc3)a0@@rZYk+_5IB%ypVHC)xBh$|_F^L-CwC?jgtJow7|(s` NDmpicuHSp|e*oE%l-U3P literal 0 HcmV?d00001