Home freenode/#haskell: Logs Calendar

Logs: freenode/#haskell

←Prev  Next→
Page 1 .. 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 .. 5022
— 502,152 events total
2020-11-13 08:44:30 hackage lentil 1.4.1.0 - frugal issue tracker https://hackage.haskell.org/package/lentil-1.4.1.0 (fffaaa)
2020-11-13 08:44:47 × britva quits (~britva@31-10-157-156.cgn.dynamic.upc.ch) (Client Quit)
2020-11-13 08:44:56 <dminuoso> Fun fact, most of Amazons profits actually comes from AWS, not Marketplace.
2020-11-13 08:45:02 <dminuoso> They are *that* darn profitable.
2020-11-13 08:45:05 × todda7 quits (~torstein@ppp-2-84-17-169.home.otenet.gr) (Ping timeout: 240 seconds)
2020-11-13 08:45:58 → todda7 joins (~torstein@ppp-2-84-17-169.home.otenet.gr)
2020-11-13 08:46:31 <merijn> dminuoso: Well, it also depends on how much fancy stuff you buy into
2020-11-13 08:46:39 → britva joins (~britva@31-10-157-156.cgn.dynamic.upc.ch)
2020-11-13 08:47:03 <merijn> My brother just got hired at a new place after doing AWS for a long time and his first reaction was "wow...you guys are wasting a lot of money on AWS..."
2020-11-13 08:47:10 → jakov joins (~jakov@95.168.120.30)
2020-11-13 08:47:21 <merijn> People just don't wanna think about performance engineering
2020-11-13 08:47:38 <dminuoso> merijn: Sure, AWS can be done cheaply with great care. But amazon counts on you not doing that.
2020-11-13 08:47:53 <dminuoso> And if you're lazy, bare metal is just cheaper.
2020-11-13 08:47:58 <merijn> I'm gonna go out and estimate that a solid 80-90% of all workloads can *easily* be handled by a single (beefy) machine
2020-11-13 08:48:03 <dminuoso> Indeed.
2020-11-13 08:48:09 <dminuoso> That is the silver bullet.
2020-11-13 08:48:19 <merijn> "ooh, no, we've got big data"
2020-11-13 08:48:34 <merijn> Fucking BS...there's like maybe 3 companies that have big data
2020-11-13 08:48:49 <dminuoso> merijn: Much of the "cloud or kubernetes" problem space addresses problems virtually noone has.
2020-11-13 08:48:59 <merijn> The rest just have incompetent software engineers and/or are too cheap to bother
2020-11-13 08:49:01 <dminuoso> Like, kubernetes is an incredibly amazing thing
2020-11-13 08:49:13 <dminuoso> If your problem is "how do I manage hundreds of services deployed to tens of thousands of nodes"
2020-11-13 08:49:20 <dminuoso> Problems everybody has in their day jobs, right?
2020-11-13 08:49:21 <merijn> dminuoso: oh, lemme look up a quote I saw recently
2020-11-13 08:49:22 <dminuoso> ;)
2020-11-13 08:49:51 → kuribas joins (~user@ptr-25vy0iaeei1zsuhust9.18120a2.ip6.access.telenet.be)
2020-11-13 08:49:56 <merijn> dminuoso: https://twitter.com/rothgar/status/1317524239218397185
2020-11-13 08:50:16 × wroathe quits (~wroathe@c-68-54-25-135.hsd1.mn.comcast.net) (Quit: leaving)
2020-11-13 08:50:20 <dminuoso> Haha that's a good one
2020-11-13 08:50:59 <dminuoso> Its the same with "high availability"
2020-11-13 08:51:01 <merijn> All the latest hype "ecosystems" are just pointless for most people
2020-11-13 08:51:15 <dminuoso> A well maintained single node has better high availability than piss-poor "Im gonna slap HAProxy ontop of this"
2020-11-13 08:51:30 <merijn> dminuoso: It's like people adopting Hadoop
2020-11-13 08:51:50 × jakov quits (~jakov@95.168.120.30) (Quit: leaving)
2020-11-13 08:51:55 <merijn> Spoiler: 80% of hadoop solutions I've seen/encountered are *slower* than a single node solution
2020-11-13 08:52:31 <dminuoso> (not to mention that high availability is a too broad term, what does that even mean? automatic failover? do we need active/active? active/passive? Or maybe we have differences wrt to writing/reading? Or maybe we want load sharing instead?
2020-11-13 08:52:56 <kuribas> merijn: brilliant
2020-11-13 08:52:58 <dminuoso> Trying to get the automatic failover right tends to introduce a lot of moving parts
2020-11-13 08:53:19 <dminuoso> like, if you add haproxy, you again have a single point of failure, and it must detect non-operatoinality
2020-11-13 08:53:39 <kuribas> merijn: wasn't there a company that rewrote all their scala code using hadoop in pure haskell running on a single machine, and found that everything was faster?
2020-11-13 08:54:11 <dminuoso> merijn: here we strive for a particular model of load sharing and automatic failover
2020-11-13 08:54:51 <merijn> dminuoso: Good failover requires robust software, but nobody has that :p
2020-11-13 08:55:07 <merijn> kuribas: I wouldn't be surprised
2020-11-13 08:55:09 <dminuoso> if service A needs to talk to service B, we just replicate service B a few times. A then maintains a pool of possible B targets, and implements load sharing by randomizing which B target to talk to
2020-11-13 08:55:21 <kuribas> merijn: ah, it was this one: https://tech.channable.com/posts/2019-10-04-why-we-decided-to-go-for-the-big-rewrite.html
2020-11-13 08:55:22 <dminuoso> If B emits failures, we mark them as temporarily dead and remove them from the pool for a while
2020-11-13 08:55:37 <dminuoso> so the high availability comes from *engineering*
2020-11-13 08:55:42 <kuribas> merijn: ah, spark, not hadoop
2020-11-13 08:55:44 <dminuoso> not "add random software ontop of it"
2020-11-13 08:55:58 <merijn> During my internship we were asked to benchmark Apache Spark (runs on Hadoop) and GraphX (runs on Spark) and found *negative* scaling
2020-11-13 08:56:06 → Yumasi joins (~guillaume@2a01cb09b06b29ea5faf5572fb93fcc2.ipv6.abo.wanadoo.fr)
2020-11-13 08:56:09 <merijn> i.e. it got slower the more nodes you added to the cluster
2020-11-13 08:56:25 <dminuoso> merijn: The usual cost of implementing the above, about 50 lines of haskell. Reliable failover with some good load sharing!
2020-11-13 08:56:30 <merijn> kuribas: Spark runs on hadoop :p
2020-11-13 08:56:43 <dminuoso> Only requirements: stateless services!
2020-11-13 08:58:52 <kuribas> I wonder how many problems cannot be solved on a single instance.
2020-11-13 08:58:58 <kuribas> or at least subdivided easily
2020-11-13 08:59:07 → heatsink joins (~heatsink@107-136-5-69.lightspeed.sntcca.sbcglobal.net)
2020-11-13 09:00:01 × aki_ quits (~aki_@139.28.218.148) ()
2020-11-13 09:00:16 → raichoo joins (~raichoo@213.240.178.58)
2020-11-13 09:00:25 <merijn> kuribas: One of my favourite graph processing papers was just a bunch of guys going: So we bought a machine with 1 TB RAM and filled it with TBs of high speed SSDs and our graph processing framework outperforms clusters of 100s of servers
2020-11-13 09:00:32 <dminuoso> kuribas: DNS servers!
2020-11-13 09:00:53 <dminuoso> If you operate an authoritative name server, it is required to have at least two of them.
2020-11-13 09:01:28 <merijn> dminuoso: I mean, back in the early 2000s there were people hobbying on MUDs that supported hot copy-over, but major companies in 2020 can't manage to update stuff without downtime >.>
2020-11-13 09:01:30 → o1lo01ol1o joins (~o1lo01ol1@bl8-213-81.dsl.telepac.pt)
2020-11-13 09:02:28 → jsmolic joins (~jakov@95.168.121.30)
2020-11-13 09:02:41 → dbmikus__ joins (~dbmikus@cpe-76-167-86-219.natsow.res.rr.com)
2020-11-13 09:02:50 × jsmolic quits (~jakov@95.168.121.30) (Client Quit)
2020-11-13 09:03:05 <kuribas> merijn: I have been lectured many time how great our lisp platform is, in that you can hot-patch everything in real-time. But when the system goes down, it takes two hours to restart.
2020-11-13 09:03:11 <kuribas> which happened a few times already
2020-11-13 09:03:13 × justsomeguy quits (~justsomeg@unaffiliated/--/x-3805311) ()
2020-11-13 09:03:39 × heatsink quits (~heatsink@107-136-5-69.lightspeed.sntcca.sbcglobal.net) (Ping timeout: 260 seconds)
2020-11-13 09:03:44 <merijn> Wow...
2020-11-13 09:03:56 <merijn> 20+ Haskell openings in EU
2020-11-13 09:04:22 → bitmagie joins (~Thunderbi@200116b8068f010041b23aab865e3023.dip.versatel-1u1.de)
2020-11-13 09:04:26 × Sanchayan quits (~Sanchayan@2401:4900:3309:dec3:efb6:6366:cb4:cbe4) (Quit: leaving)
2020-11-13 09:04:34 <kuribas> maybe I should apply there...
2020-11-13 09:04:41 <kuribas> I wonder if they allow 4/5
2020-11-13 09:05:00 <dminuoso> merijn: facebook has hot code replacement with haskell :P
2020-11-13 09:05:21 <dminuoso> but in general, updating a live system is complicated
2020-11-13 09:05:50 <dminuoso> Especially if there's multiple parts, because you cant generally execute an update atomically
2020-11-13 09:06:01 <merijn> kuribas: THat's what negotiation is for ;)
2020-11-13 09:06:16 <dminuoso> What does 4/5 mean?
2020-11-13 09:06:23 <kuribas> dminuoso: 4 day workweek
2020-11-13 09:06:25 <merijn> dminuoso: I assume 32 hour work week
2020-11-13 09:06:30 <kuribas> yes
2020-11-13 09:06:39 <dminuoso> ah
2020-11-13 09:07:01 <merijn> Looks like they're mostly in Scandinavia so, that's probably negotiable
2020-11-13 09:07:13 × dbmikus__ quits (~dbmikus@cpe-76-167-86-219.natsow.res.rr.com) (Ping timeout: 256 seconds)
2020-11-13 09:08:50 <dminuoso> merijn: Oh and regarding our earlier discussion with "single node", my current project involves a "servant api + postgres" server.
2020-11-13 09:09:20 <dminuoso> Since all but one systems only need read-only, we've decided to duplicate the servant+postgres onto every sever that needs to talk to my API
2020-11-13 09:09:36 <dminuoso> So there's one master servant+postgres, and everything else receives postgres streaming replicatoin
2020-11-13 09:10:39 × pavonia quits (~user@unaffiliated/siracusa) (Quit: Bye!)
2020-11-13 09:10:41 × britva quits (~britva@31-10-157-156.cgn.dynamic.upc.ch) (Quit: This computer has gone to sleep)
2020-11-13 09:10:56 <dminuoso> so if any node fails, everyone else can still talk to their local API running on a local database
2020-11-13 09:11:19 <dminuoso> that way, we remove the network dependency from the service, and have higher resilience of any single node failing
2020-11-13 09:11:52 <dminuoso> (this service is a mission critical piece in our infrastructure, we just cant take the chance of a singular node going down at the wrong time(
2020-11-13 09:12:27 → jsmolic joins (~jakov@95.168.121.30)
2020-11-13 09:12:44 → britva joins (~britva@2a02:aa13:7240:2980:bc4b:509a:98e6:5bb0)
2020-11-13 09:13:03 → invaser joins (~Thunderbi@31.148.23.125)
←Prev  Next→
Page 1 .. 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 .. 5022

All times are in UTC.