Logs: liberachat/#haskell
| 2025-09-11 12:42:07 | → | trickard_ joins (~trickard@cpe-54-98-47-163.wireline.com.au) |
| 2025-09-11 12:42:34 | → | jreicher joins (~user@user/jreicher) |
| 2025-09-11 12:43:35 | × | petrichor quits (~jez@user/petrichor) (Quit: ZNC 1.10.1 - https://znc.in) |
| 2025-09-11 12:45:14 | × | humasect quits (~humasect@dyn-192-249-132-90.nexicom.net) (Remote host closed the connection) |
| 2025-09-11 12:46:07 | → | humasect joins (~humasect@dyn-192-249-132-90.nexicom.net) |
| 2025-09-11 12:48:02 | <kqr> | I have a CPU bound loop that is embarassingly parallel. What would be the modern way to split that up into equal chunks and run each of them on a separate core? I tried naïvely using forConcurrently from the async library, but that seemed like it ended up causing more contention and context switches because the evaluation went a lot slower with it. |
| 2025-09-11 12:48:17 | <kqr> | (I did compile with -threaded and run with -N2 to test.) |
| 2025-09-11 12:49:25 | <merijn> | kqr: Yeah, forConcurrently for CPU bound tasks spawns a lot more threads than useful |
| 2025-09-11 12:51:04 | × | humasect quits (~humasect@dyn-192-249-132-90.nexicom.net) (Ping timeout: 248 seconds) |
| 2025-09-11 12:51:08 | <merijn> | kqr: Fortunately for you, I had exactly this issue, approximately a decade ago and I got so tired of reinventing the wheel for myself that this exists: https://hackage-content.haskell.org/package/broadcast-chan-0.3.0/docs/BroadcastChan.html#g:4 |
| 2025-09-11 12:51:29 | <kqr> | Since these are very short-lived tasks I also suspect it'd benefit from setting up the plumbing first and then using light operations to throw tasks at workers. |
| 2025-09-11 12:51:59 | <merijn> | kqr: That's pretty much what the linked code does :p |
| 2025-09-11 12:52:14 | → | lbseale joins (~quassel@user/ep1ctetus) |
| 2025-09-11 12:52:42 | <L29Ah> | kqr: https://hackage.haskell.org/package/parallel-io-0.3.5/docs/Control-Concurrent-ParallelIO-Local.html maybe |
| 2025-09-11 12:52:48 | <merijn> | Ignore the large dependency list shown by hackage, it's not great at showing dependencies for multi-lib packages. The core library I linked has a transitive dependency tree of 3 (including base and transformers) |
| 2025-09-11 12:55:37 | <kqr> | merijn, if I understand it correctly, the parFoldMap would be setting up the plumbing afresh each time it is called, so if I use that I would want it to wrap a larger piece of work. Alternatively I could spawn a few workers and hand them BroadcastChan handles to receive work from if I want to schedule smaller pieces of work at a time? |
| 2025-09-11 12:59:23 | × | segfaultfizzbuzz quits (~segfaultf@23-93-74-222.fiber.dynamic.sonic.net) (Ping timeout: 250 seconds) |
| 2025-09-11 12:59:26 | → | simplystuart joins (~simplystu@c-75-75-152-164.hsd1.pa.comcast.net) |
| 2025-09-11 13:02:20 | → | petrichor joins (~jez@user/petrichor) |
| 2025-09-11 13:05:58 | × | mange quits (~mange@user/mange) (Quit: Zzz...) |
| 2025-09-11 13:06:13 | → | AndreiDuma joins (~AndreiDum@user/AndreiDuma) |
| 2025-09-11 13:08:26 | <kqr> | L29Ah, I tested that first because it was easiest to adapt the code to. Takes 2.5× longer than single-threaded when I run it with two threads. It seems like it doesn't pin the processes to threads and just go with it, but swap them around or something, because none of the cores are particularly utilised while it runs in parallel. |
| 2025-09-11 13:09:28 | <L29Ah> | sounds strange, it worked well for me |
| 2025-09-11 13:10:12 | <L29Ah> | -O2 -threaded -rtsopts "-with-rtsopts -N" |
| 2025-09-11 13:10:57 | <kqr> | https://entropicthoughts.com/pastes/Main-e34639.hs.html That's the code, compiled -threaded and run +RTS -N2. I don't think I'm using it wrong. |
| 2025-09-11 13:11:37 | × | AndreiDuma quits (~AndreiDum@user/AndreiDuma) (Quit: My Mac has gone to sleep. ZZZzzz…) |
| 2025-09-11 13:12:59 | × | vanishingideal quits (~vanishing@user/vanishingideal) (Ping timeout: 248 seconds) |
| 2025-09-11 13:13:27 | → | Enrico63 joins (~Enrico63@2a0b:e541:10d0:0:9efc:e8ff:fe24:3213) |
| 2025-09-11 13:20:06 | <merijn> | kqr: the conduit library lets you stream jobs into parallel workers |
| 2025-09-11 13:20:48 | <merijn> | kqr: That's what I did, use conduit to stream task descriptions into workers, then reassemble after processing in parallel |
| 2025-09-11 13:28:43 | <merijn> | kqr: For large inputs I'm not at all surprised it's slower |
| 2025-09-11 13:29:07 | <merijn> | Since it just immediately forks of N (where N is your input size) forkIO threads, then has them fighting over the available RTS capabilities to finish |
| 2025-09-11 13:29:19 | → | segfaultfizzbuzz joins (~segfaultf@23-93-74-222.fiber.dynamic.sonic.net) |
| 2025-09-11 13:29:23 | <merijn> | Which is fine if they're largely blocking, but pointless for CPU bound work |
| 2025-09-11 13:30:27 | <merijn> | kqr: The parMapM/parMapM_ in the conduit library are pretty nice if you want to stream a large number of jobs: https://hackage-content.haskell.org/package/broadcast-chan-0.3.0/docs/BroadcastChan-Conduit.html#v:parMapM |
| 2025-09-11 13:42:05 | <L29Ah> | kqr: lgtm, i use it similarily in hyborg |
| 2025-09-11 13:42:51 | <L29Ah> | merijn: are you sure you don't have funny thread-blocking things like unsafePerformIO in your workload? |
| 2025-09-11 13:43:07 | <L29Ah> | or those "safe" FFI calls |
| 2025-09-11 13:43:26 | <L29Ah> | er kqr |
| 2025-09-11 13:43:47 | <merijn> | unsafePerformIO itself should be fine |
| 2025-09-11 13:43:51 | <merijn> | and safe FFI calls too |
| 2025-09-11 13:44:21 | <L29Ah> | unsafePerformIO stops all the other threads until finished IIRC |
| 2025-09-11 13:44:33 | <merijn> | That's definitely not true |
| 2025-09-11 13:45:24 | <merijn> | The closest thing to that that is true is unsafe foreign calls block garbage collection (and can thus block other capabilities if they need to GC) |
| 2025-09-11 13:46:03 | <merijn> | but unsafe foreign calls are a far cry from unsafePerformIO |
| 2025-09-11 13:47:59 | × | segfaultfizzbuzz quits (~segfaultf@23-93-74-222.fiber.dynamic.sonic.net) (Ping timeout: 260 seconds) |
| 2025-09-11 13:48:18 | <tomsmeding> | IIRC parallel evaluation of a thunk that is an unsafePerformIO will lock |
| 2025-09-11 13:48:42 | <tomsmeding> | but that is redundant parallel evaluation of _one_ thunk, which is not helpful anyway |
| 2025-09-11 13:48:50 | <merijn> | tomsmeding: Yes |
| 2025-09-11 13:49:14 | <merijn> | something with blackholing/grey holing |
| 2025-09-11 13:49:19 | <merijn> | And I always forget which is which |
| 2025-09-11 13:49:24 | <tomsmeding> | ¯\_(ツ)_/¯ |
| 2025-09-11 13:50:17 | <tomsmeding> | merijn: unsafePerformIO does something more special than normal blackholing though, which is what all thunks do to catch <<loop>> |
| 2025-09-11 13:50:22 | <L29Ah> | it is helpful if it is a thing like a static array read |
| 2025-09-11 13:50:27 | <tomsmeding> | there is this noDuplicate# call where I don't know what it does |
| 2025-09-11 13:50:34 | <L29Ah> | while waiting for a lock is very expensive compared to the read |
| 2025-09-11 13:50:41 | <merijn> | tomsmeding: Yeah, <<loop>> is blackholing, I think the parallel evaluation thing is greyholing |
| 2025-09-11 13:50:43 | → | CiaoSen joins (~Jura@2a02:8071:64e1:da0:5a47:caff:fe78:33db) |
| 2025-09-11 13:50:48 | <tomsmeding> | L29Ah: a static array read should probably use unsafeDupablePerformIO if you care about parallel efficiency |
| 2025-09-11 13:50:58 | <tomsmeding> | because that doesn't do the additional noDuplicate stuff |
| 2025-09-11 13:51:01 | <tomsmeding> | ah |
| 2025-09-11 13:51:19 | <L29Ah> | https://github.com/haskell/unix/issues/157 yes but people don't believe |
| 2025-09-11 13:51:28 | <L29Ah> | anyway that's just my guess re kqr's lack of performance |
| 2025-09-11 13:52:07 | <merijn> | L29Ah: I mean, the "forces the program to run single-threaded" is still wrong |
| 2025-09-11 13:52:08 | <L29Ah> | mayhaps lock trashing would result in worse-than-single-thread performance |
| 2025-09-11 13:52:21 | <L29Ah> | yes it is imprecise, sorry :] |
| 2025-09-11 13:52:25 | <merijn> | L29Ah: And tbh, I don't think unsafePerformIO has any meaningful impact on any of the operations that unix does |
| 2025-09-11 13:52:49 | <L29Ah> | it does when you have millions of files to poke in your backup program |
| 2025-09-11 13:53:48 | <tomsmeding> | L29Ah: I think the unix maintainers will be more eager to make the changes you request if you come with an actual benchmark :) |
| 2025-09-11 13:54:18 | <tomsmeding> | because "unsafePerformIO makes your code run single-threaded" is simply false, but it may well be that due to some other effects, that ends up being the case in your particular use-case |
| 2025-09-11 14:06:50 | → | humasect joins (~humasect@dyn-192-249-132-90.nexicom.net) |
| 2025-09-11 14:08:22 | <EvanR> | not all IO actions even do I/O |
| 2025-09-11 14:10:28 | <EvanR> | or access shared resources |
| 2025-09-11 14:13:26 | → | SlackCoder joins (~SlackCode@remote.nationalgallery.org.ky) |
| 2025-09-11 14:15:13 | × | humasect quits (~humasect@dyn-192-249-132-90.nexicom.net) (Ping timeout: 250 seconds) |
| 2025-09-11 14:16:57 | → | Sgeo joins (~Sgeo@user/sgeo) |
| 2025-09-11 14:17:19 | → | mari-estel joins (~mari-este@user/mari-estel) |
| 2025-09-11 14:33:06 | → | inline joins (~inline@ip-005-146-196-014.um05.pools.vodafone-ip.de) |
| 2025-09-11 14:35:23 | × | jbalint quits (~jbalint@syn-071-090-116-115.res.spectrum.com) (Quit: Bye!) |
| 2025-09-11 14:39:10 | × | CiaoSen quits (~Jura@2a02:8071:64e1:da0:5a47:caff:fe78:33db) (Ping timeout: 244 seconds) |
| 2025-09-11 14:39:20 | → | emperori joins (~emperori@223.187.122.81) |
| 2025-09-11 14:43:54 | × | inline quits (~inline@ip-005-146-196-014.um05.pools.vodafone-ip.de) (Read error: Connection reset by peer) |
| 2025-09-11 14:43:57 | × | emperori quits (~emperori@223.187.122.81) (Read error: Connection reset by peer) |
| 2025-09-11 14:44:21 | → | humasect joins (~humasect@dyn-192-249-132-90.nexicom.net) |
| 2025-09-11 14:45:24 | → | Guest56 joins (~Guest56@140.235.141.169) |
| 2025-09-11 14:45:31 | × | Guest56 quits (~Guest56@140.235.141.169) (Client Quit) |
| 2025-09-11 14:45:55 | → | afu1101 joins (~afu1101@140.235.141.167) |
| 2025-09-11 14:46:29 | × | afu1101 quits (~afu1101@140.235.141.167) (Client Quit) |
| 2025-09-11 14:46:59 | → | mari81549 joins (~mari-este@user/mari-estel) |
| 2025-09-11 14:47:28 | × | lortabac quits (~lortabac@2a01:e0a:541:b8f0:55ab:e185:7f81:54a4) (Quit: WeeChat 4.5.2) |
| 2025-09-11 14:47:49 | → | inline joins (~inline@ip-005-146-196-014.um05.pools.vodafone-ip.de) |
| 2025-09-11 14:48:37 | × | humasect quits (~humasect@dyn-192-249-132-90.nexicom.net) (Ping timeout: 258 seconds) |
| 2025-09-11 14:48:52 | × | merijn quits (~merijn@77.242.116.146) (Ping timeout: 265 seconds) |
| 2025-09-11 14:48:58 | × | mari-estel quits (~mari-este@user/mari-estel) (Read error: Connection reset by peer) |
| 2025-09-11 14:49:50 | → | merijn joins (~merijn@77.242.116.146) |
| 2025-09-11 14:59:47 | → | Lycurgus joins (~juan@user/Lycurgus) |
| 2025-09-11 15:00:53 | × | califax quits (~califax@user/califx) (Remote host closed the connection) |
| 2025-09-11 15:01:08 | → | califax joins (~califax@user/califx) |
| 2025-09-11 15:06:57 | × | fp quits (~Thunderbi@wireless-86-50-140-161.open.aalto.fi) (Ping timeout: 248 seconds) |
| 2025-09-11 15:07:13 | × | petrichor quits (~jez@user/petrichor) (Ping timeout: 250 seconds) |
All times are in UTC.