[{"data":1,"prerenderedAt":568},["ShallowReactive",2],{"\u002Fen\u002Fpost\u002F2020\u002F05\u002Fquickstart-apache-spark-on-kubernetes":3},{"id":4,"title":5,"author":6,"body":7,"createdAt":552,"description":553,"extension":554,"meta":555,"navigation":558,"path":559,"seo":560,"slug":561,"stem":562,"tags":563,"__hash__":567},"posts\u002Fposts\u002F2020\u002F05\u002Fquickstart-apache-spark-on-kubernetes.md","Quickstart: Apache Spark on Kubernetes",null,{"type":8,"value":9,"toc":543},"minimark",[10,15,20,68,87,96,126,135,155,163,167,188,202,229,243,252,256,260,263,330,337,347,351,368,374,378,381,387,399,405,415,423,438,442,445,451,454,481,484,492,499,505,508,514,517,523,526,532,536],[11,12,14],"h1",{"id":13},"introduction","Introduction",[16,17,19],"h2",{"id":18},"the-apache-spark-operator-for-kubernetes","The Apache Spark Operator for Kubernetes",[21,22,23,24,28,29,33,34,37,38,45,50,55,56,61,62,67],"p",{},"Since its launch in 2014 by Google, Kubernetes has gained a lot of\npopularity along with Docker itself and since 2016 has become the ",[25,26,27],"em",{},"de\nfacto Container Orchestrator",", established as a market standard.\nHaving cloud-managed versions available in ",[30,31,32],"strong",{},"all"," the ",[25,35,36],{},"major Clouds",".\n",[39,40,44],"a",{"href":41,"rel":42},"https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002F",[43],"nofollow","[1]",[39,46,49],{"href":47,"rel":48},"https:\u002F\u002Faws.amazon.com\u002Feks\u002F",[43],"[2]",[39,51,54],{"href":52,"rel":53},"https:\u002F\u002Fdocs.microsoft.com\u002Fen-us\u002Fazure\u002Faks\u002F",[43],"[3]"," (including\n",[39,57,60],{"href":58,"rel":59},"https:\u002F\u002Fwww.digitalocean.com\u002Fproducts\u002Fkubernetes\u002F",[43],"Digital Ocean"," and\n",[39,63,66],{"href":64,"rel":65},"https:\u002F\u002Fwww.alibabacloud.com\u002Fproduct\u002Fkubernetes",[43],"Alibaba",").",[21,69,70,71,74,75,80,81,86],{},"With this popularity came various implementations and ",[25,72,73],{},"use-cases"," of\nthe orchestrator, among them the execution of ",[39,76,79],{"href":77,"rel":78},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftutorials\u002Fstateful-application\u002F",[43],"Stateful\napplications","\nincluding ",[39,82,85],{"href":83,"rel":84},"https:\u002F\u002Fvitess.io\u002Fzh\u002Fdocs\u002Fget-started\u002Fkubernetes\u002F",[43],"databases using containers",".",[21,88,89,90,95],{},"What would be the motivation to host an orchestrated database? That's\na great question. But let's focus on the ",[39,91,94],{"href":92,"rel":93},"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fspark-on-k8s-operator\u002Fblob\u002Fmaster\u002Fdocs\u002Fdesign.md",[43],"Spark Operator","\nrunning workloads on Kubernetes.",[21,97,98,99,104,105,108,109,114,115,120,121,86],{},"A native Spark Operator ",[39,100,103],{"href":101,"rel":102},"https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fkubernetes\u002Fissues\u002F34377",[43],"idea came out","\nin 2016, before that you couldn't run Spark jobs natively except\nsome ",[25,106,107],{},"hacky alternatives",", like ",[39,110,113],{"href":111,"rel":112},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2016\u002F03\u002Fusing-spark-and-zeppelin-to-process-big-data-on-kubernetes\u002F",[43],"running Apache Zeppelin","\ninside Kubernetes or creating your ",[39,116,119],{"href":117,"rel":118},"https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fexamples\u002Ftree\u002Fmaster\u002Fstaging\u002Fspark",[43],"Apache Spark cluster inside\nKubernetes (from the official Kubernetes organization on GitHub)","\nreferencing the ",[39,122,125],{"href":123,"rel":124},"http:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fspark-standalone.html",[43],"Spark workers in Stand-alone mode",[21,127,128,129,134],{},"However, the native execution would be far more interesting for taking\nadvantage of ",[39,130,133],{"href":131,"rel":132},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Fkube-scheduler\u002F",[43],"Kubernetes Scheduler","\nresponsible for taking action of allocating resources, giving\nelasticity and an simpler interface to manage Apache Spark workloads.",[21,136,137,138,143,144,149,150,86],{},"Considering that, ",[39,139,142],{"href":140,"rel":141},"https:\u002F\u002Fissues.apache.org\u002Fjira\u002Fbrowse\u002FSPARK-18278",[43],"Apache Spark Operator development got attention",",\nmerged and released into ",[39,145,148],{"href":146,"rel":147},"https:\u002F\u002Fspark.apache.org\u002Freleases\u002Fspark-release-2-3-0.html",[43],"Spark version 2.3.0","\nlaunched in ",[39,151,154],{"href":152,"rel":153},"https:\u002F\u002Fspark.apache.org\u002Fnews\u002Findex.html",[43],"February, 2018",[21,156,157,158],{},"If you're eager for reading more regarding the Apache Spark proposal,\nyou can head to the ",[39,159,162],{"href":160,"rel":161},"https:\u002F\u002Fdocs.google.com\u002Fdocument\u002Fd\u002F1_bBzOZ8rKiOSjQg78DXOA3ZBIo_KkDJjqxVuq0yXdew\u002Fedit#heading=h.9bhogel14x0y",[43],"design document published in Google Docs.",[16,164,166],{"id":165},"why-kubernetes","Why Kubernetes?",[21,168,169,170,175,176,180,184,86],{},"As companies are currently seeking to ",[39,171,174],{"href":172,"rel":173},"https:\u002F\u002Fwww.cio.com\u002Farticle\u002F3211428\u002Fwhat-is-digital-transformation-a-necessary-disruption.html",[43],"reinvent themselves through the\nwidely spoken digital transformation","\nin order for them to be competitive and, above all, to survive in an\nincreasingly dynamic market, it is common to see approaches that\ninclude Big Data, Artificial Intelligence and Cloud Computing\n",[39,177,44],{"href":178,"rel":179},"https:\u002F\u002Fwww.zdnet.com\u002Farticle\u002Fhow-to-use-cloud-computing-and-big-data-to-support-digital-transformation\u002F",[43],[39,181,49],{"href":182,"rel":183},"https:\u002F\u002Fdigitalhealth.london\u002Fcloud-big-data-ai-lead-nhs-digital-transformation\u002F",[43],[39,185,54],{"href":186,"rel":187},"https:\u002F\u002Fwww.ibm.com\u002Fblogs\u002Fcloud-computing\u002F2018\u002F11\u002F05\u002Fguiding-framework-digital-transformation-garage\u002F",[43],[21,189,190,191,196,197,86],{},"An interesting comparison between the benefits of using Cloud Computing in the\ncontext of Big Data instead of On-premises' servers can be read at ",[39,192,195],{"href":193,"rel":194},"https:\u002F\u002Fdatabricks.com\u002Fblog\u002F2017\u002F05\u002F31\u002Ftop-5-reasons-for-choosing-s3-over-hdfs.html",[43],"Databricks\nblog",",\nwhich is the company ",[39,198,201],{"href":199,"rel":200},"https:\u002F\u002Fwww.washingtonpost.com\u002Fnews\u002Fthe-switch\u002Fwp\u002F2016\u002F06\u002F09\u002Fthis-is-where-the-real-action-in-artificial-intelligence-takes-place\u002F",[43],"founded by the creators of Apache Spark",[21,203,204,205,210,211,216,217,222,223,228],{},"As we see a widespread adoption of Cloud Computing (even by companies\nthat would be able to afford the hardware and run on-premises), we\nnotice that most of these Cloud implementations don't have an ",[39,206,209],{"href":207,"rel":208},"https:\u002F\u002Fhadoop.apache.org\u002F",[43],"Apache\nHadoop"," since the Data Teams (BI\u002FData\nScience\u002FAnalytics) increasingly choose to use tools like ",[39,212,215],{"href":213,"rel":214},"https:\u002F\u002Fcloud.google.com\u002Fbigquery\u002F",[43],"Google\nBigQuery"," or ",[39,218,221],{"href":219,"rel":220},"https:\u002F\u002Faws.amazon.com\u002Fredshift\u002F",[43],"AWS Redshift",".\nTherefore, it doesn't make sense to spin-up a Hadoop with the only intention to\nuse ",[39,224,227],{"href":225,"rel":226},"https:\u002F\u002Fhortonworks.com\u002Fapache\u002Fyarn\u002F",[43],"YARN"," as the resources manager.",[21,230,231,232,216,237,242],{},"An alternative is the use of Hadoop cluster providers such as ",[39,233,236],{"href":234,"rel":235},"https:\u002F\u002Fcloud.google.com\u002Fdataproc",[43],"Google\nDataProc",[39,238,241],{"href":239,"rel":240},"https:\u002F\u002Faws.amazon.com\u002Femr\u002F",[43],"AWS EMR","\nfor the creation of ephemeral clusters. Just to name a few options.",[21,244,245,246,251],{},"To better understand the design of Spark Operator, the doc from ",[39,247,250],{"href":248,"rel":249},"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fspark-on-k8s-operatoR\u002Fblob\u002Fmaster\u002Fdocs\u002Fdesign.md#the-crd-controller",[43],"GCP on GitHub","\nis a no-brainer.",[11,253,255],{"id":254},"lets-get-hands-on","Let's get hands-on!",[16,257,259],{"id":258},"warming-up-the-engine","Warming up the engine",[21,261,262],{},"Now that the word has been spread, let's get our hands on it to show\nthe engine running. For that, let's use:",[264,265,266,281,289,302],"ul",{},[267,268,269,274,275,280],"li",{},[39,270,273],{"href":271,"rel":272},"https:\u002F\u002Fwww.docker.com\u002F",[43],"Docker"," as the container engine for\nKubernetes ",[39,276,279],{"href":277,"rel":278},"https:\u002F\u002Fdocs.docker.com\u002Finstall\u002F",[43],"(installation guide)",";",[267,282,283,284,288],{},"Minikube ",[39,285,279],{"href":286,"rel":287},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Ftools\u002Finstall-minikube\u002F",[43],"\nto facilitate the provisioning of the Kubernetes (yes, it will be\na local execution);",[267,290,291,292,296,297,86],{},"For interaction with the Kubernetes API it is necessary to have\n",[293,294,295],"code",{},"kubectl"," installed, ",[39,298,301],{"href":299,"rel":300},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Ftools\u002Finstall-kubectl\u002F",[43],"if you don't have it, follow instructions\nhere",[267,303,304,305],{},"a compiled version of Apache Spark larger than 2.3.0.\n",[306,307,308,321],"ol",{},[267,309,310,311,316,317,320],{},"you can either compile ",[39,312,315],{"href":313,"rel":314},"https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark",[43],"source code",",\nwhich will took ",[25,318,319],{},"some hours"," to finish, or",[267,322,323,324,329],{},"download a compiled version ",[39,325,328],{"href":326,"rel":327},"https:\u002F\u002Fspark.apache.org\u002Fdownloads.html",[43],"here","\n(recommended).",[21,331,332,333,336],{},"Once the necessary tools are installed, it's necessary to\ninclude Apache Spark path in ",[293,334,335],{},"PATH"," environment variable, to ease the\ninvocation of Apache Spark executables. Simply run:",[338,339,345],"pre",{"className":340,"code":342,"language":343,"meta":344},[341],"language-bash","export PATH=${PATH}:\u002Fpath\u002Fto\u002Fapache-spark-X.Y.Z\u002Fbin\n","bash","",[293,346,342],{"__ignoreMap":344},[16,348,350],{"id":349},"creating-the-minikube-cluster","Creating the Minikube \"cluster\"",[21,352,353,354,357,358,363,364,367],{},"At last, to have a Kubernetes \"cluster\" we will start a ",[293,355,356],{},"minikube","\nwith the intention of running an example from ",[39,359,362],{"href":360,"rel":361},"https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark\u002Fblob\u002Fmaster\u002Fexamples\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002Fexamples\u002FSparkPi.scala",[43],"Spark\nrepository","\ncalled ",[293,365,366],{},"SparkPi"," just as a demonstration.",[338,369,372],{"className":370,"code":371,"language":343,"meta":344},[341],"minikube start --cpus=2 \\\n    --memory=4g\n",[293,373,371],{"__ignoreMap":344},[16,375,377],{"id":376},"building-the-docker-image","Building the Docker image",[21,379,380],{},"Let's use the Minikube Docker daemon to not depend on an external registry (and\nonly generate Docker image layers on the VM, facilitating garbage disposal\nlater). Minikube has a wrapper that makes our life easier:",[338,382,385],{"className":383,"code":384,"language":343,"meta":344},[341],"eval $(minikube docker-env)\n",[293,386,384],{"__ignoreMap":344},[21,388,389,390,395,396,398],{},"After having the daemon environment variables configured, we need a\nDocker image to run the jobs. There is a ",[39,391,394],{"href":392,"rel":393},"https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark\u002Fblob\u002Fmaster\u002Fbin\u002Fdocker-image-tool.sh",[43],"shell script in the Spark\nrepository","\nto help with this. Considering that our ",[293,397,335],{}," was properly\nconfigured, just run:",[338,400,403],{"className":401,"code":402,"language":343,"meta":344},[341],"docker-image-tool.sh -m -t latest build\n",[293,404,402],{"__ignoreMap":344},[21,406,407,410,411,414],{},[25,408,409],{},"FYI:"," The ",[293,412,413],{},"-m"," parameter here indicates a minikube build.",[21,416,417,418,86],{},"Let's take the highway to execute SparkPi, using the same command\nthat would be used for a Hadoop Spark cluster ",[39,419,422],{"href":420,"rel":421},"https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsubmitting-applications.html",[43],"spark-submit",[21,424,425,426,431,432,437],{},"However, Spark Operator supports defining jobs in the \"Kubernetes\ndialect\" using ",[39,427,430],{"href":428,"rel":429},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fextend-kubernetes\u002Fapi-extension\u002Fcustom-resources\u002F",[43],"CRD",",\n",[39,433,436],{"href":434,"rel":435},"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fspark-on-k8s-operator\u002Ftree\u002Fmaster\u002Fexamples",[43],"here are some examples"," - for later.",[11,439,441],{"id":440},"fire-in-the-hole","Fire in the hole!",[21,443,444],{},"Mid the gap between the Scala version and .jar when you're\nparameterizing with your Apache Spark version:",[338,446,449],{"className":447,"code":448,"language":343,"meta":344},[341],"spark-submit --master k8s:\u002F\u002Fhttps:\u002F\u002F$(minikube ip):8443 \\\n    --deploy-mode cluster \\\n    --name spark-pi \\\n    --class org.apache.spark.examples.SparkPi \\\n    --conf spark.executor.instances=2 \\\n    --executor-memory 1024m \\\n    --conf spark.kubernetes.container.image=spark:latest \\\n    local:\u002F\u002F\u002Fopt\u002Fspark\u002Fexamples\u002Fjars\u002Fspark-examples_2.11-X.Y.Z.jar # here\n",[293,450,448],{"__ignoreMap":344},[21,452,453],{},"What's new is:",[264,455,456,475],{},[267,457,458,461,462,465,466,469,470,280],{},[293,459,460],{},"--master",": Accepts a prefix ",[293,463,464],{},"k8s:\u002F\u002F"," in the URL, for the\nKubernetes master API endpoint, exposed by the command\n",[293,467,468],{},"https:\u002F\u002F$(minikube ip):8443",". BTW, in case you want to\nknow, it's a ",[39,471,474],{"href":472,"rel":473},"https:\u002F\u002Fwww.gnu.org\u002Fsoftware\u002Fbash\u002Fmanual\u002Fhtml_node\u002FCommand-Substitution.html",[43],"shell command substitution",[267,476,477,480],{},[293,478,479],{},"--conf spark.kubernetes.container.image=",": Configures the Docker\nimage to run in Kubernetes.",[21,482,483],{},"Sample output:",[338,485,490],{"className":486,"code":488,"language":489},[487],"language-text","...\n\n19\u002F08\u002F22 11:59:09 INFO LoggingPodStatusWatcherImpl: State changed,\nnew state: pod name: spark-pi-1566485909677-driver namespace: default\nlabels: spark-app-selector -> spark-20477e803e7648a59e9bcd37394f7f60,\nspark-role -> driver pod uid: c789c4d2-27c4-45ce-ba10-539940cccb8d\ncreation time: 2019-08-22T14:58:30Z service account name: default\nvolumes: spark-local-dir-1, spark-conf-volume, default-token-tj7jn\nnode name: minikube start time: 2019-08-22T14:58:30Z container\nimages: spark:docker phase: Succeeded status:\n[ContainerStatus(containerID=docker:\u002F\u002Fe044d944d2ebee2855cd2b993c62025d\n6406258ef247648a5902bf6ac09801cc, image=spark:docker,\nimageID=docker:\u002F\u002Fsha256:86649110778a10aa5d6997d1e3d556b35454e9657978f3\na87de32c21787ff82f, lastState=ContainerState(running=null,\nterminated=null, waiting=null, additionalProperties={}),\nname=spark-kubernetes-driver, ready=false, restartCount=0,\nstate=ContainerState(running=null,\nterminated=ContainerStateTerminated(containerID=docker:\u002F\u002Fe044d944d2ebe\ne2855cd2b993c62025d6406258ef247648a5902bf6ac09801cc, exitCode=0,\nfinishedAt=2019-08-22T14:59:08Z, message=null, reason=Completed,\nsignal=null, startedAt=2019-08-22T14:58:32Z,\nadditionalProperties={}), waiting=null, additionalProperties={}),\nadditionalProperties={})]\n\n19\u002F08\u002F22 11:59:09 INFO LoggingPodStatusWatcherImpl: Container final\nstatuses: Container name: spark-kubernetes-driver Container image:\nspark:docker Container state: Terminated Exit code: 0\n","text",[293,491,488],{"__ignoreMap":344},[21,493,494,495,498],{},"To see the job result (and the whole execution) we can run a\n",[293,496,497],{},"kubectl logs"," passing the name of the driver pod as a parameter:",[338,500,503],{"className":501,"code":502,"language":343,"meta":344},[341],"kubectl logs $(kubectl get pods | grep 'spark-pi.*-driver')\n",[293,504,502],{"__ignoreMap":344},[21,506,507],{},"Which brings the output (omitted some entries), similar to:",[338,509,512],{"className":510,"code":511,"language":489},[487],"...\n19\u002F08\u002F22 14:59:08 INFO TaskSetManager: Finished task 1.0 in stage 0.0\n(TID 1) in 52 ms on 172.17.0.7 (executor 1) (2\u002F2)\n19\u002F08\u002F22 14:59:08 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose\ntasks have all completed, from pool19\u002F08\u002F22 14:59:08 INFO\nDAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in\n0.957 s\n19\u002F08\u002F22 14:59:08 INFO DAGScheduler: Job 0 finished: reduce at\nSparkPi.scala:38, took 1.040608 s Pi is roughly 3.138915694578473\n19\u002F08\u002F22 14:59:08 INFO SparkUI: Stopped Spark web UI at\nhttp:\u002F\u002Fspark-pi-1566485909677-driver-svc.default.svc:4040\n19\u002F08\u002F22 14:59:08 INFO KubernetesClusterSchedulerBackend: Shutting\ndown all executors\n19\u002F08\u002F22 14:59:08 INFO\nKubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking\neach executor to shut down\n19\u002F08\u002F22 14:59:08 WARN ExecutorPodsWatchSnapshotSource: Kubernetes\nclient has been closed (this is expected if the application is\nshutting down.)\n19\u002F08\u002F22 14:59:08 INFO MapOutputTrackerMasterEndpoint:\nMapOutputTrackerMasterEndpoint stopped!\n19\u002F08\u002F22 14:59:08 INFO MemoryStore: MemoryStore cleared\n19\u002F08\u002F22 14:59:08 INFO BlockManager: BlockManager stopped\n19\u002F08\u002F22 14:59:08 INFO BlockManagerMaster: BlockManagerMaster stopped\n19\u002F08\u002F22 14:59:08 INFO\nOutputCommitCoordinator$OutputCommitCoordinatorEndpoint:\nOutputCommitCoordinator stopped!\n19\u002F08\u002F22 14:59:08 INFO SparkContext: Successfully stopped SparkContext\n19\u002F08\u002F22 14:59:08 INFO ShutdownHookManager: Shutdown hook called\n19\u002F08\u002F22 14:59:08 INFO ShutdownHookManager: Deleting directory\n\u002Ftmp\u002Fspark-aeadc6ba-36aa-4b7e-8c74-53aa48c3c9b2\n19\u002F08\u002F22 14:59:08 INFO ShutdownHookManager: Deleting directory\n\u002Fvar\u002Fdata\u002Fspark-084e8326-c8ce-4042-a2ed-75c1eb80414a\u002Fspark-ef8117bf-90\nd0-4a0d-9cab-f36a7bb18910\n...\n",[293,513,511],{"__ignoreMap":344},[21,515,516],{},"The result appears in:",[338,518,521],{"className":519,"code":520,"language":489},[487],"19\u002F08\u002F22 14:59:08 INFO DAGScheduler: Job 0 finished: reduce at\nSparkPi.scala:38, took 1.040608 s Pi is roughly 3.138915694578473\n",[293,522,520],{"__ignoreMap":344},[21,524,525],{},"Finally, let's delete the VM that Minikube generates, to clean up the\nenvironment (unless you want to keep playing with it):",[338,527,530],{"className":528,"code":529,"language":343,"meta":344},[341],"minikube delete\n",[293,531,529],{"__ignoreMap":344},[16,533,535],{"id":534},"last-words","Last words",[21,537,538,539,542],{},"I hope your curiosity got ",[25,540,541],{},"sparked"," and some ideas for further\ndevelopment have raised for your Big Data workloads. If you have any\ndoubt or suggestion, don't hesitate to share on the comment section.",{"title":344,"searchDepth":544,"depth":544,"links":545},2,[546,547,548,549,550,551],{"id":18,"depth":544,"text":19},{"id":165,"depth":544,"text":166},{"id":258,"depth":544,"text":259},{"id":349,"depth":544,"text":350},{"id":376,"depth":544,"text":377},{"id":534,"depth":544,"text":535},"2020-05-21T23:00:57+02:00","Using Apache Spark Operator in Kubernetes to streamline your Big Data workflows with a cloud-native approach without relying on a Hadoop cluster.","md",{"subtitle":556,"image":557},"Running Apache Spark Operator on Kubernetes","\u002Fimages\u002Fcontent\u002F2019\u002Fbig-load-of-containers.png",true,"\u002Fposts\u002F2020\u002F05\u002Fquickstart-apache-spark-on-kubernetes",{"title":5,"description":553},"quickstart-apache-spark-on-kubernetes","posts\u002F2020\u002F05\u002Fquickstart-apache-spark-on-kubernetes",[564,565,566],"data pipelines","kubernetes","tutorials","VXxVM6rUzN-01z99mq_SGPe_8MbD3d8sidyg7Trfa0E",1778441744188]