Logs: liberachat/#haskell
| 2021-08-16 10:37:01 | → | haritz joins (~hrtz@user/haritz) |
| 2021-08-16 10:37:54 | × | merijn quits (~merijn@83-160-49-249.ip.xs4all.nl) (Ping timeout: 268 seconds) |
| 2021-08-16 10:38:28 | → | merijn joins (~merijn@83-160-49-249.ip.xs4all.nl) |
| 2021-08-16 10:41:55 | × | Erutuon quits (~Erutuon@user/erutuon) (Ping timeout: 258 seconds) |
| 2021-08-16 10:46:17 | × | vysn quits (~vysn@user/vysn) (Ping timeout: 245 seconds) |
| 2021-08-16 10:46:42 | × | jgeerds quits (~jgeerds@55d45555.access.ecotel.net) (Ping timeout: 245 seconds) |
| 2021-08-16 10:47:36 | → | dextaa joins (~DV@user/dextaa) |
| 2021-08-16 10:53:30 | → | nsilv joins (~nsilv@host-79-17-175-58.retail.telecomitalia.it) |
| 2021-08-16 10:56:46 | → | Obo joins (~roberto@78.77.166.185) |
| 2021-08-16 10:56:52 | × | Codaraxis_ quits (~Codaraxis@user/codaraxis) (Ping timeout: 258 seconds) |
| 2021-08-16 10:57:46 | → | wallymathieu joins (~wallymath@81-234-151-21-no94.tbcn.telia.com) |
| 2021-08-16 10:58:32 | <maerwald> | language interop should get much more focus instead of polluting our type system more |
| 2021-08-16 11:00:26 | <merijn> | maerwald: Well, find me a source who will fund work on "the next Haskell" to focus more on runtime guarantees for exceptions, concurrency support, and better foreign integration, and I'll get right on that ;) |
| 2021-08-16 11:00:51 | <maerwald> | doesn't sound flashy enough, give me dependent types instead |
| 2021-08-16 11:01:04 | → | alx741 joins (~alx741@186.178.108.32) |
| 2021-08-16 11:01:46 | → | Codaraxis_ joins (~Codaraxis@user/codaraxis) |
| 2021-08-16 11:01:50 | × | burnsidesLlama quits (~burnsides@dhcp168-025.wadham.ox.ac.uk) (Remote host closed the connection) |
| 2021-08-16 11:02:22 | → | burnsidesLlama joins (~burnsides@dhcp168-025.wadham.ox.ac.uk) |
| 2021-08-16 11:02:29 | <gehmehgeh> | hmm, I'm looking for a "filter"-function that throws away the input list (or other structure) while it is filtering. I've been applying "filter" on a very large list (that I don't need, I only need the filter result) |
| 2021-08-16 11:02:39 | <gehmehgeh> | said large list is growing and growing |
| 2021-08-16 11:02:45 | × | wallymathieu quits (~wallymath@81-234-151-21-no94.tbcn.telia.com) (Quit: My MacBook has gone to sleep. ZZZzzz…) |
| 2021-08-16 11:02:58 | <tomsmeding> | filter does exactly that |
| 2021-08-16 11:03:05 | <tomsmeding> | are you observing different behaviour? |
| 2021-08-16 11:03:15 | <gehmehgeh> | hmm, interesting. Maybe I'm misunderstanding my own code. Let me see |
| 2021-08-16 11:03:27 | <tomsmeding> | happens because of laziness :p |
| 2021-08-16 11:03:34 | <tomsmeding> | how are you generating that large list |
| 2021-08-16 11:03:44 | <tomsmeding> | monadically, perhaps? |
| 2021-08-16 11:06:40 | × | burnsidesLlama quits (~burnsides@dhcp168-025.wadham.ox.ac.uk) (Ping timeout: 240 seconds) |
| 2021-08-16 11:06:41 | × | nsilv quits (~nsilv@host-79-17-175-58.retail.telecomitalia.it) (Ping timeout: 248 seconds) |
| 2021-08-16 11:07:39 | <arahael> | tomsmeding: Tonight I migrated to libarchive. :) Was very easy, though I couldn't find the docs, so I just read the github repo for the libarchive haskell binding instead. :) |
| 2021-08-16 11:08:06 | <tomsmeding> | arahael: the penultimate version on hackage does have docs generated :p |
| 2021-08-16 11:08:07 | <tomsmeding> | but yay! |
| 2021-08-16 11:08:25 | <arahael> | tomsmeding: Ah, but I couldn't see links on https://hackage.haskell.org/package/libarchive |
| 2021-08-16 11:08:37 | <tomsmeding> | arahael: https://hackage.haskell.org/package/libarchive-3.0.2.1 |
| 2021-08-16 11:08:39 | <arahael> | I see the penultimate version does indeed have them - didn't think to check. |
| 2021-08-16 11:09:03 | <tomsmeding> | yeah I happen to know that that sometimes happens (don't know how) |
| 2021-08-16 11:09:23 | <arahael> | Interesting - well, now I know too! :) |
| 2021-08-16 11:10:01 | → | jtomas joins (~jtomas@233.red-83-34-2.dynamicip.rima-tde.net) |
| 2021-08-16 11:11:03 | <tomsmeding> | something went wrong in that latest patch release anyway, seeing as the changelog file on hackage is still the old one too |
| 2021-08-16 11:13:34 | × | derelict quits (~derelict@user/derelict) (Ping timeout: 272 seconds) |
| 2021-08-16 11:15:49 | <gehmehgeh> | tomsmeding: what do you mean by "monadically" in that context? |
| 2021-08-16 11:15:57 | <gehmehgeh> | tomsmeding: Could you give an example? |
| 2021-08-16 11:15:59 | <gehmehgeh> | :) |
| 2021-08-16 11:16:13 | <arahael> | tomsmeding: Interesting, well, I'm very pleased at how easy it was to rip out the Tar and replace it with libArchive. :) |
| 2021-08-16 11:16:46 | <tomsmeding> | arahael: no more filename truncation I hope? |
| 2021-08-16 11:16:46 | <Drew[m]1> | gehmehgeh: Even if you force the input list beforehand so that the whole thing is sitting in memory, The garbage collector should be able to figure out that the big list isn't used any more if it *can't* be used any more, and should GC it as appropriate. |
| 2021-08-16 11:16:49 | <Drew[m]1> | Maybe you need to finagle your code so that the garbage collector is suitably convinced that the big list could not be used after filtering: |
| 2021-08-16 11:16:49 | <Drew[m]1> | such as defining it in the local scope of a function that returns the filtered list, so once the function returns the filtered list, the |
| 2021-08-16 11:16:49 | <Drew[m]1> | big list goes out of scope. |
| 2021-08-16 11:17:15 | <arahael> | tomsmeding: None! |
| 2021-08-16 11:17:32 | <arahael> | tomsmeding: Curiously, when I unpacked and re-tarred using gnu tar, I was still getting filename truncation! |
| 2021-08-16 11:17:41 | <arahael> | tomsmeding: But libarchive is perfect. And faster, to boot. |
| 2021-08-16 11:17:57 | × | Codaraxis_ quits (~Codaraxis@user/codaraxis) (Ping timeout: 245 seconds) |
| 2021-08-16 11:18:36 | <gehmehgeh> | Drew[m]1: hm, thanks |
| 2021-08-16 11:18:45 | <arahael> | Ooh, fascinating, I didn't check the memory usage before, but I'm now using a *TB* of memory! :D |
| 2021-08-16 11:18:55 | <arahael> | Only 2.2g RES, though. |
| 2021-08-16 11:19:06 | <merijn> | RES is all tha matters, anyway |
| 2021-08-16 11:19:18 | <arahael> | But yes, a full 1024 GB of virtual memory. Which is fascinating. |
| 2021-08-16 11:19:28 | <tomsmeding> | % let build1 n = n : build1 (n + 1) |
| 2021-08-16 11:19:28 | <yahb> | tomsmeding: |
| 2021-08-16 11:19:32 | <tomsmeding> | % let build2 n = (:) <$> pure n <*> build2 (n + 1) |
| 2021-08-16 11:19:32 | <yahb> | tomsmeding: |
| 2021-08-16 11:19:37 | <arahael> | merijn: Indeed. I believe virtual memory includes mmap'ped pages and all that, too. |
| 2021-08-16 11:19:37 | <tomsmeding> | % take 10 (build1 1) |
| 2021-08-16 11:19:37 | <yahb> | tomsmeding: [1,2,3,4,5,6,7,8,9,10] |
| 2021-08-16 11:19:44 | <merijn> | arahael: Eh |
| 2021-08-16 11:19:50 | <tomsmeding> | % fmap (take 10) (build2 1 :: Maybe [Int]) |
| 2021-08-16 11:19:55 | <yahb> | tomsmeding: [Timed out] |
| 2021-08-16 11:19:59 | <merijn> | arahael: GHC compiled programs as of 8.0ish *always* use 1 TB virtual memory, because top is dumb |
| 2021-08-16 11:20:00 | <tomsmeding> | gehmehgeh: ^ |
| 2021-08-16 11:20:09 | <arahael> | merijn: Oh? Curious! |
| 2021-08-16 11:20:27 | <merijn> | arahael: Top reports *reserved* (but unused) address space as part of VIRT usage |
| 2021-08-16 11:20:28 | <tomsmeding> | merijn: is that because top is dumb or because ghc really does that |
| 2021-08-16 11:20:35 | <tomsmeding> | ah |
| 2021-08-16 11:20:44 | <merijn> | tomsmeding: GHC uses the OS as intended |
| 2021-08-16 11:20:47 | <arahael> | merijn: Eh? That's dumb, indeed. |
| 2021-08-16 11:20:56 | <tomsmeding> | merijn: nobody was claiming GHC didn't do that |
| 2021-08-16 11:21:05 | <merijn> | Basically, if C allocations and GHC get interleaved in memory is a hassle |
| 2021-08-16 11:21:11 | <merijn> | It messes with the "copy & compact" |
| 2021-08-16 11:21:27 | <merijn> | So, it's preferable that all Haskell allocations are in adjacent addresses |
| 2021-08-16 11:21:57 | <merijn> | To stop the OS/C from littering allocations in the middle, the runtime preemptively reserves 1 TB of virtual memory and dynamically grows that *actual* allocation within that |
| 2021-08-16 11:21:58 | × | bitdex quits (~bitdex@gateway/tor-sasl/bitdex) (Quit: = "") |
| 2021-08-16 11:22:12 | <gehmehgeh> | tomsmeding: Thanks for the example, I'm not sure I understand it why this gresults in memory problems |
| 2021-08-16 11:22:14 | <merijn> | Which means that any Haskell program with less than 1TB RES will have its Haskell data adjacent |
| 2021-08-16 11:22:22 | <merijn> | Without worrying about C, etc. |
| 2021-08-16 11:22:25 | <tomsmeding> | which is kind of the whole idea of linux' virtual memory overallocation scheme, that you can do this |
| 2021-08-16 11:22:41 | <merijn> | But since top doesn't treat virtual reservations separately from virtual *mappings* it always reports 1TB |
| 2021-08-16 11:22:42 | <arahael> | Interesting. I'm actually really surprised I'm only using 2.2g because I'm doing stupid things like "uncompress an entire 3.3g file into a bytestring, then use libarchive to load THAT as a .tar file, and then put ALL the files in that as a HashMap". |
| 2021-08-16 11:22:51 | <tomsmeding> | gehmehgeh: not sure if this is the problem that you have, but I just wanted to highlight this in case it is |
| 2021-08-16 11:23:18 | → | Codaraxis_ joins (~Codaraxis@user/codaraxis) |
| 2021-08-16 11:23:29 | <arahael> | Eh, only 1.3 G, that explains it slightly, but still, 1.3g .tar.gz -> 2.2g actual memory isn't so bad. |
| 2021-08-16 11:24:06 | <merijn> | I mean, GHC's GC design is *really* elegant if you only care about efficiency/throughput :) |
| 2021-08-16 11:24:34 | <merijn> | It's only when you care about latency that things can become iffy |
| 2021-08-16 11:24:39 | <tomsmeding> | (my illustration was: if you're building a list in a strict monad (~all of them), the full tail will have to be evaluated at runtime before anything can be returned. Indeed, what if at some point, build2 would return Nothing? Then there's no list to be returned!) |
| 2021-08-16 11:25:05 | <tomsmeding> | (and the same holds if you replace Maybe with IO, which is more likely to be the case) |
| 2021-08-16 11:25:20 | → | pfurla_ joins (~pfurla@ool-3f8fcb0f.dyn.optonline.net) |
| 2021-08-16 11:25:47 | × | Codaraxis_ quits (~Codaraxis@user/codaraxis) (Remote host closed the connection) |
| 2021-08-16 11:27:02 | <arahael> | merijn: I'm hoping that the latency generally doesn't get /that/ bad. |
| 2021-08-16 11:28:26 | <merijn> | arahael: Usually not |
| 2021-08-16 11:28:40 | × | pfurla quits (~pfurla@ool-3f8fcb0f.dyn.optonline.net) (Ping timeout: 240 seconds) |
All times are in UTC.