Logs: liberachat/#haskell
| 2021-08-24 06:59:12 | <arahael> | It's basically a google protocol buffers web api, but a naive one at that. |
| 2021-08-24 06:59:17 | <dminuoso> | timCF: It's a natural consequence from the construction. |
| 2021-08-24 06:59:18 | <arahael> | (Instead of JSON) |
| 2021-08-24 06:59:23 | <dminuoso> | timCF: To see why, consider the following: |
| 2021-08-24 06:59:43 | <dminuoso> | timCF: `traverse (traverse f) [[1,2,3], [2,3,4]]]` |
| 2021-08-24 06:59:55 | <dminuoso> | Which traverse touches the "inner" numbers", and which traverse works on the lists? |
| 2021-08-24 07:00:36 | <arahael> | Rembane_: I'm just wondering if there's any way of, eg, remotely loading it in some sort of real-time debugger or log viewer where I can see a report of open files. |
| 2021-08-24 07:00:47 | <timCF> | dminuoso: 2nd works on numbers I ghuess |
| 2021-08-24 07:01:03 | <arahael> | Rembane_: Such that I could test for any leaks when I do a curl or something. |
| 2021-08-24 07:01:03 | <dminuoso> | timCF: Can you try and assert this for yourself? |
| 2021-08-24 07:01:14 | <dminuoso> | timCF: Want to ensure that you yourself are sure of this fact. |
| 2021-08-24 07:01:15 | <Rembane_> | arahael: If you're running Linux you can get that by the command lsof -i |
| 2021-08-24 07:02:24 | <Rembane_> | arahael: lsof -ai -p PIDOFTHEPROCESS |
| 2021-08-24 07:02:32 | → | d0ku joins (~d0ku@178.43.56.75.ipv4.supernova.orange.pl) |
| 2021-08-24 07:02:40 | × | tzh quits (~tzh@c-24-21-73-154.hsd1.wa.comcast.net) (Quit: zzz) |
| 2021-08-24 07:03:29 | → | gehmehgeh joins (~user@user/gehmehgeh) |
| 2021-08-24 07:04:14 | <arahael> | Rembane_: That just gives: familyser 585947 arahael 3u IPv4 184973807 0t0 TCP *:hbci (LISTEN) |
| 2021-08-24 07:04:25 | × | Kaiepi quits (~Kaiepi@156.34.44.192) (Ping timeout: 252 seconds) |
| 2021-08-24 07:05:05 | <timCF> | dminuoso: yeah, I did tried it here https://replit.com/languages/haskell like this `traverse (traverse print) [[1,2,3], [2,3,4]]` and got column of numbers |
| 2021-08-24 07:05:26 | <Rembane_> | arahael: I wonder if your server process starts a sub process to hold the connections, does it seem to do that or is it just one process? |
| 2021-08-24 07:05:52 | × | merijn quits (~merijn@83-160-49-249.ip.xs4all.nl) (Ping timeout: 250 seconds) |
| 2021-08-24 07:06:56 | <arahael> | Rembane_: I don't think so. I mean, there's the cabal process itself, but that then just starts the actual server. |
| 2021-08-24 07:07:39 | tomsmeding | . o O ( pstree PIDOFPROCESS ) |
| 2021-08-24 07:09:14 | <dminuoso> | timCF: Alright. Now consider: |
| 2021-08-24 07:09:25 | <dminuoso> | (traverse . traverse) f [[1,2,3], [2,3,4]] |
| 2021-08-24 07:09:29 | <dminuoso> | Do you agree that this is the same thing? |
| 2021-08-24 07:09:51 | <arahael> | tomsmeding: That seems to identify that I have the right PID, at least. |
| 2021-08-24 07:09:54 | → | dhouthoo joins (~dhouthoo@178-117-36-167.access.telenet.be) |
| 2021-08-24 07:10:02 | × | gehmehgeh quits (~user@user/gehmehgeh) (Quit: Leaving) |
| 2021-08-24 07:10:13 | <tomsmeding> | arahael: and, apparently, shows that there are no subprocesses you didn't know about :) |
| 2021-08-24 07:11:38 | <arahael> | tomsmeding: Indeed, it doesn't! Actually I just restarted my client application, and then that lsof command showed a billion lines. (No, not a billion, a great many), but now a few seconds later when I ran it again, there's just that line I showed already. |
| 2021-08-24 07:11:52 | → | Boomerang joins (~Boomerang@xd520f68c.cust.hiper.dk) |
| 2021-08-24 07:12:08 | <arahael> | So I for a very short time had many, many lines that looked like this: |
| 2021-08-24 07:12:10 | <arahael> | familyser 585947 arahael 53u IPv4 185217813 0t0 TCP pihole:hbci->192.168.1.105:52946 (ESTABLISHED) |
| 2021-08-24 07:12:29 | × | azeem quits (~azeem@dynamic-adsl-94-34-33-6.clienti.tiscali.it) (Ping timeout: 250 seconds) |
| 2021-08-24 07:12:34 | <arahael> | (The local server also happens to run pihole, hence that appearing there) |
| 2021-08-24 07:13:25 | <timCF> | dminuoso: yeah, result seems the same |
| 2021-08-24 07:13:37 | <arahael> | Just counted them with wc -l: It outputs 52 lines. |
| 2021-08-24 07:13:52 | <tomsmeding> | that's not too many |
| 2021-08-24 07:13:54 | → | azeem joins (~azeem@176.201.15.223) |
| 2021-08-24 07:14:13 | × | Erutuon quits (~Erutuon@user/erutuon) (Ping timeout: 250 seconds) |
| 2021-08-24 07:14:30 | <tomsmeding> | try adding -n to the lsof command, less weird port names |
| 2021-08-24 07:14:41 | <timCF> | dminuoso: but I think this time first in the line "traverse" is responsible for applying `f` to numbers |
| 2021-08-24 07:15:43 | <tomsmeding> | timCF: (traverse1 . traverse2) f x = (\y -> traverse1 (traverse2 y)) f x = traverse1 (traverse2 f) x |
| 2021-08-24 07:15:53 | <arahael> | tomsmeding: Good tip. Still, this doesn't seem to be indicating any problems - zero lines where there are indeed zero connections - and presumeably, one line for each connection. |
| 2021-08-24 07:16:22 | <arahael> | Yet, in a few days, at some random point (haven't seen a pattern), it'll just suddenly die. |
| 2021-08-24 07:16:24 | <tomsmeding> | arahael: kind of sounds like in normal operation there aren't too many FDs open, but when the problem arises, it suddenly spikes or something |
| 2021-08-24 07:16:35 | <arahael> | tomsmeding: Could be, yes. |
| 2021-08-24 07:18:39 | → | hannessteffenhag joins (~hannesste@ip4d14ffc8.dynamic.kabel-deutschland.de) |
| 2021-08-24 07:18:54 | <arahael> | Going to spam the localhost with a while true; connect loop. |
| 2021-08-24 07:19:15 | <tomsmeding> | arahael: what I would do is set up a process that just does 'while true; do echo "$(date) $(lsof -ni PIDOFPROCESS | wc -l)"; sleep 120; done >>log.txt' |
| 2021-08-24 07:19:37 | <tomsmeding> | and you'll find whether it's a gradual rise or something more sudden :) |
| 2021-08-24 07:20:39 | <arahael> | Hmm, odd. I couldn't kill the server that way, using `while true; curl...&; done`, ended up having lines like: curl: c(u7r)l :C o(u7l)d nC'otu lcdonn'nte ccto ntnoe cste rtvoe rs |
| 2021-08-24 07:20:58 | <tomsmeding> | :D |
| 2021-08-24 07:20:59 | <timCF> | tomsmeding: hmm, you are right. In my mind I've always pronounced `.` not as "compose" but as "after" when I'm writing or reading some code. Because "after" is much easier to imagine than "compose". But seems like in some cases this is misleading. |
| 2021-08-24 07:21:01 | <arahael> | Which means after a while I was getting "Couldn't connect to server". |
| 2021-08-24 07:21:06 | <Rembane_> | arahael: I like the level of chaos in that. :D |
| 2021-08-24 07:21:10 | <Axman6> | concurr is hard cy |
| 2021-08-24 07:21:11 | <arahael> | Rembane_: :D |
| 2021-08-24 07:21:15 | <tomsmeding> | timCF: "after" is definitely correct |
| 2021-08-24 07:21:25 | <dminuoso> | timCF: Now, the next bit is this: |
| 2021-08-24 07:21:31 | <dminuoso> | `traverse` *is* a valid traversal |
| 2021-08-24 07:21:33 | × | lavaman quits (~lavaman@98.38.249.169) (Remote host closed the connection) |
| 2021-08-24 07:21:37 | <tomsmeding> | it's applying traverse1 after traverse2 to f |
| 2021-08-24 07:21:41 | <dminuoso> | It's the traversal that targets *each* element in a given structure. |
| 2021-08-24 07:21:54 | <timCF> | Yeah |
| 2021-08-24 07:22:00 | <dminuoso> | And `traverse . traverse` is a composed traversal. |
| 2021-08-24 07:22:06 | <dminuoso> | We can even use this with lens: |
| 2021-08-24 07:22:16 | <dminuoso> | % toListOf (traverse . traverse) [[1,2,3,], [2,3,4]] |
| 2021-08-24 07:22:17 | <yahb> | dminuoso: ; <interactive>:11:40: error: parse error on input `]' |
| 2021-08-24 07:22:23 | <dminuoso> | % toListOf (traverse . traverse) [[1,2,3], [2,3,4]] |
| 2021-08-24 07:22:23 | <yahb> | dminuoso: [1,2,3,2,3,4] |
| 2021-08-24 07:22:24 | → | jtomas joins (~jtomas@233.red-83-34-2.dynamicip.rima-tde.net) |
| 2021-08-24 07:23:04 | <dminuoso> | c.f.: |
| 2021-08-24 07:23:08 | <dminuoso> | % :i Traversal |
| 2021-08-24 07:23:09 | <yahb> | dminuoso: type role ReifiedTraversal representational nominal representational nominal; type ReifiedTraversal :: * -> * -> * -> * -> *; newtype ReifiedTraversal s t a b = Traversal {...}; -- Defined in `Control.Lens.Reified'; type Traversal :: * -> * -> * -> * -> *; type Traversal s t a b = forall (f :: * -> *). Applicative f => (a -> f b) -> s -> f t; -- Defined in `Control.Lens.Type' |
| 2021-08-24 07:23:30 | <dminuoso> | type Traversal s t a b = forall (f :: * -> *). Applicative f => (a -> f b) -> s -> f t; |
| 2021-08-24 07:23:33 | <dminuoso> | % :t traverse |
| 2021-08-24 07:23:33 | <yahb> | dminuoso: (Traversable t, Applicative f) => (a -> f b) -> t a -> f (t b) |
| 2021-08-24 07:23:45 | <arahael> | tomsmeding: It seems that wc -l will usually report in just '2'. I guess I have to watch it, and take note of the actual error message to cut&paste, I only remember that it was running out of files somehow, and that it suggested I run it in a threaded configuration instead to work around it. |
| 2021-08-24 07:23:49 | <dminuoso> | timCF: A lens is something similar to this, except the constraint is Functor, not Applicative. |
| 2021-08-24 07:24:04 | <dminuoso> | timCF: If you care, I can give you a set of simple examples to explore this further |
| 2021-08-24 07:24:12 | <dminuoso> | Well. Tasks really |
| 2021-08-24 07:24:18 | <arahael> | Ooh, I've got it to slowly accumulate more and more connections. Nice. (Found a slow API, which I'm polling as fast as I can) |
| 2021-08-24 07:24:24 | <tomsmeding> | arahael: yeah my intent with that while-loop was to let that running for a few days until it crashes |
| 2021-08-24 07:26:10 | <arahael> | tomsmeding: I think I'll get it to crash in a few moments. Possibly minutes. :) I've realised that my testing with the client where there's only 52 connections was problematic because I've implemented etags, so now it's too fast if it's already cached. |
| 2021-08-24 07:26:21 | <arahael> | It's up to 550 files now. |
| 2021-08-24 07:26:32 | <arahael> | Or at least, 550-2 files. |
| 2021-08-24 07:26:33 | <tomsmeding> | "yay"? |
| 2021-08-24 07:27:11 | <arahael> | Yeah. |
| 2021-08-24 07:27:18 | <arahael> | Stupid optimisations. ;) |
| 2021-08-24 07:27:50 | <arahael> | I can tell that the server is clearly single-threaded, processing connections very very sequentially. |
| 2021-08-24 07:28:10 | <timCF> | dminuoso: Thanks! Seems like explanation why lens compose "backwards" is more complex than I can handle right now) Probably I need to read some FPComplete tutorials and implement it myself to really understand why it works this way) |
| 2021-08-24 07:28:12 | <tomsmeding> | arahael: seems that if the server runs out of FDs, then the client should also be using quite a significant number of connections by then |
| 2021-08-24 07:28:25 | <arahael> | That's my thinking, yeah. |
| 2021-08-24 07:28:26 | <dminuoso> | 09:14:41 timCF | dminuoso: but I think this time first in the line "traverse" is responsible for applying `f` to numbers |
| 2021-08-24 07:28:35 | <dminuoso> | timCF: If you understood this, you already understood why lenses compose "backwards" |
| 2021-08-24 07:28:43 | <dminuoso> | Nothing further to be understood. |
| 2021-08-24 07:28:45 | <timCF> | dminuoso: :) |
| 2021-08-24 07:29:03 | <arahael> | tomsmeding: Just passed a thousand connections! |
All times are in UTC.