Logs: liberachat/#haskell
| 2021-07-01 20:37:39 | <geekosaur> | @where paste |
| 2021-07-01 20:37:39 | <lambdabot> | Help us help you: please paste full code, input and/or output at e.g. https://paste.tomsmeding.com |
| 2021-07-01 20:37:59 | <dminuoso> | ixlun: What cost centers are generated? Did you use all-functions? |
| 2021-07-01 20:38:34 | × | fizbin quits (~fizbin@c-73-33-197-160.hsd1.nj.comcast.net) (Remote host closed the connection) |
| 2021-07-01 20:38:47 | <tomsmeding> | ixlun: does the (huge) call stack trace below the top part of the profiling output tell anything interesting? |
| 2021-07-01 20:38:49 | <davean> | injectLeftovers is pretty clear in what it does - don't have left overs, read just what you need? Is there a reason you're sending too much data down to consumers? |
| 2021-07-01 20:39:07 | <tomsmeding> | not sure how using a streaming library messes up those call stacks |
| 2021-07-01 20:39:12 | × | alphabeta quits (~kilolympu@cpc92710-cmbg20-2-0-cust265.5-4.cable.virginm.net) (Ping timeout: 272 seconds) |
| 2021-07-01 20:39:32 | <davean> | tomsmeding: its just functions? You might have to understand how streaming libraries are implimented though to understand what they mean |
| 2021-07-01 20:39:45 | → | fizbin joins (~fizbin@c-73-33-197-160.hsd1.nj.comcast.net) |
| 2021-07-01 20:39:49 | <dminuoso> | Indeed |
| 2021-07-01 20:39:53 | <ixlun> | dminuoso: I'm not sure, I used the `--enable-profiling` cabal option. |
| 2021-07-01 20:40:09 | <tomsmeding> | I thought there may be sufficient CPS going on that the callstack doesn't make sense anymore, but then I haven't used streaming libraries yet so in particular not with a profiler :p |
| 2021-07-01 20:40:41 | <chris_> | https://paste.tomsmeding.com/sSE5bwAK |
| 2021-07-01 20:40:53 | <davean> | tomsmeding: I mean I don't find CPS ruins callstacks, but I also understand the CPS, so your mileage might vary? |
| 2021-07-01 20:41:01 | <ixlun> | davean: Indeed, I think that's the issue. Since the whole conduit is a function, the translation means that it's hard to figure out which part of the conduit is causing issues. |
| 2021-07-01 20:41:07 | × | fizbin quits (~fizbin@c-73-33-197-160.hsd1.nj.comcast.net) (Remote host closed the connection) |
| 2021-07-01 20:41:19 | <dminuoso> | ixlun: Also add `--profiling-detail=all-functions`, then you might get more granular data. |
| 2021-07-01 20:41:29 | <tomsmeding> | davean: I was just guessing, you're likely right :) |
| 2021-07-01 20:42:02 | <dminuoso> | ixlun: At any rate, judging from an initial look at conduit, as long as you're just using conduit directly, it seems the only reasonable explanation is that you're only consuming little input. Are you perhaps left with too much.. leftovers at the end of your pipe? |
| 2021-07-01 20:42:04 | <davean> | tomsmeding: it does mean you have to know EVERY name in the internals of the library, but we're not talking large libraries |
| 2021-07-01 20:42:25 | <dminuoso> | ixlun: If you re-run with --profiling-detail=all-functions, perhaps you can share the output? |
| 2021-07-01 20:42:26 | <davean> | dminuoso: they can happen at every step |
| 2021-07-01 20:42:35 | <tomsmeding> | chris_: let the function take an extra parameter, e.g. "increment", where you add "increment" instead of 2, and recursively call with increment+2? |
| 2021-07-01 20:43:03 | <davean> | dminuoso: so every monadic bind for example |
| 2021-07-01 20:43:04 | <dminuoso> | davean: Yes, but if the cost center is right on `injectLeftovers`, then it's a matter of understanding where it's mostly called. |
| 2021-07-01 20:43:10 | <koz> | For InstanceD, I get the parts of an instance declaration represented by the 'Maybe Overlap' and 'Cxt', but not the 'Type'. |
| 2021-07-01 20:43:13 | <dminuoso> | davean: And injeftLeftovers is not called on monadic bind |
| 2021-07-01 20:43:36 | <koz> | In the example in the docs, does the 'Type' correspond to 'Show [w]', or just '[w]'? |
| 2021-07-01 20:43:40 | <ixlun> | dminuoso: Yep, running the profile now. |
| 2021-07-01 20:43:45 | <chris_> | tomsmeding: i could do that, but then 1 of the params is redundant in a way |
| 2021-07-01 20:43:49 | <dminuoso> | davean: Though, I think we're a bit at a cross roads. |
| 2021-07-01 20:43:50 | <koz> | Docs here: https://hackage.haskell.org/package/template-haskell-2.17.0.0/docs/Language-Haskell-TH.html#t:Dec |
| 2021-07-01 20:44:05 | <tomsmeding> | chris_: then write a wrapper function that just calls the function you just wrote with 2 :) |
| 2021-07-01 20:44:16 | × | machinedgod quits (~machinedg@24.105.81.50) (Ping timeout: 272 seconds) |
| 2021-07-01 20:44:24 | <chris_> | ahhh ye that's prob what i want |
| 2021-07-01 20:44:28 | <tomsmeding> | you can even hide the inner function by putting it in a 'where' block below the wrapper function |
| 2021-07-01 20:44:39 | <tomsmeding> | this wrapper pattern is quite common in haskell |
| 2021-07-01 20:44:55 | × | MQ-17J quits (~MQ-17J@d14-69-206-129.try.wideopenwest.com) (Ping timeout: 265 seconds) |
| 2021-07-01 20:45:03 | <davean> | dminuoso: well it depends on what things he's using - I'm saying it can show up at any, not that it does show up at any |
| 2021-07-01 20:45:08 | <dminuoso> | Right |
| 2021-07-01 20:45:13 | <chris_> | that's what i was thinking of ye intitially, shall i google haskell wrapper pattern? you have an example? |
| 2021-07-01 20:45:28 | <tomsmeding> | not sure if that's the right name to be honest |
| 2021-07-01 20:45:30 | <dminuoso> | davean: Hence, I asked for --profiling-detail=all-functions, hopefully this will give better cost centers. :) |
| 2021-07-01 20:45:52 | <davean> | Why am I helping with conduit though? I ran away from that, I'm going back to machines where things make sense :-p |
| 2021-07-01 20:46:23 | <davean> | If I want pushback to be a thing, I'll use a stack! |
| 2021-07-01 20:46:33 | <ixlun> | dminuoso: https://termbin.com/tn6c |
| 2021-07-01 20:46:36 | <dminuoso> | Okay... are you comparing conduit to a stack now? :< |
| 2021-07-01 20:46:38 | × | GIANTWORLDKEEPER quits (~pjetcetal@128-71-13-182.broadband.corbina.ru) (Quit: EXIT) |
| 2021-07-01 20:46:49 | <davean> | dminuoso: Huh? |
| 2021-07-01 20:46:56 | <tomsmeding> | chris_: https://hackage.haskell.org/package/base-4.15.0.0/docs/src/GHC-List.html#zipWith |
| 2021-07-01 20:47:00 | <Cale> | chris_: What if you were to make a list of the intervals to start with, and then zipWith that with the ds? |
| 2021-07-01 20:47:02 | <davean> | dminuoso: pushback is a stack operation |
| 2021-07-01 20:47:17 | <Cale> | oh, you're already talking about this :) |
| 2021-07-01 20:47:41 | <davean> | dminuoso: https://hackage.haskell.org/package/machines-0.7.2/docs/Data-Machine-Stack.html |
| 2021-07-01 20:47:48 | <tomsmeding> | Cale: no, I was just showing an example of this wrapper pattern thing :p |
| 2021-07-01 20:47:49 | <dminuoso> | Yes I understand.. |
| 2021-07-01 20:47:52 | <Cale> | ah |
| 2021-07-01 20:47:53 | <tomsmeding> | and accidentally gave precisely zipWith |
| 2021-07-01 20:48:02 | <dminuoso> | davean: Lets just drop it, I think we got our wires crossed here. |
| 2021-07-01 20:48:03 | <davean> | dminuoso: if the entirety of the initial input is the dataset, pushback is stack push, and all reads are stack pop |
| 2021-07-01 20:48:07 | <davean> | dminuoso: ok |
| 2021-07-01 20:48:09 | <davean> | I'm confused |
| 2021-07-01 20:48:24 | <dminuoso> | oh |
| 2021-07-01 20:48:28 | <dminuoso> | Now I see what you mean |
| 2021-07-01 20:48:36 | <dminuoso> | Odd, but okay |
| 2021-07-01 20:48:46 | <davean> | Its litterly the stack operational semantics |
| 2021-07-01 20:48:54 | <davean> | which allows you to compose it |
| 2021-07-01 20:49:00 | <davean> | which allows you to control how it happens and where |
| 2021-07-01 20:49:32 | <davean> | The implicity and inability to control it is one of the major pain points IME in optimizing streaming |
| 2021-07-01 20:50:08 | <Cale> | chris_: So, you'd have something like f ds interval = foldr (:+:) (rest 0) $ zipWith (\d n -> note d (trans n (A,2)) [interval, 2*interval ..] |
| 2021-07-01 20:50:14 | <dminuoso> | But isnt a conduit more like a fifo? You cant continuously push on a stack while you're popping and preserve the input order.. |
| 2021-07-01 20:50:34 | <dminuoso> | I must be misunderstanding something here |
| 2021-07-01 20:50:48 | <davean> | dminuoso: no, it isn't, not internally - consider how "take 50" would be implimented on a string |
| 2021-07-01 20:51:00 | <davean> | now consider how it would be implimented on Text |
| 2021-07-01 20:51:29 | <davean> | With text you'd get a chunk that reached >= 50 charactures |
| 2021-07-01 20:51:44 | <davean> | now, if you got the next characture, if you didn't stack push it, you'd lose the over-read data |
| 2021-07-01 20:52:00 | <davean> | do { drop 50; head } |
| 2021-07-01 20:52:03 | <davean> | what does head return? |
| 2021-07-01 20:52:20 | <davean> | if the semantics were to differ between Text and String you'd have an issue |
| 2021-07-01 20:53:02 | <tomsmeding> | (this sounds to me like you have a queue with a putback operation at the taking end, not necessarily that you now have a stack) |
| 2021-07-01 20:53:12 | <ixlun> | Now my conduit does have a place where it takes one input value and outputs 15 values for each value. Would those be classed as leftovers? |
| 2021-07-01 20:53:13 | <davean> | take/drop are somewhat simple examples, but consider parsers, which might have to read to see if they should consume |
| 2021-07-01 20:53:14 | <tomsmeding> | (but /me is out of my depth here) |
| 2021-07-01 20:53:14 | → | shapr joins (~user@pool-108-28-144-11.washdc.fios.verizon.net) |
| 2021-07-01 20:53:40 | <davean> | tomsmeding: you (virtually) populate the entire stack with the entire data that will be streamed initially, and then you pop and push |
| 2021-07-01 20:54:13 | <tomsmeding> | oooh you don't see the initial population as stack operations |
| 2021-07-01 20:54:28 | <davean> | Its like setting the tape in a turing machine |
| 2021-07-01 20:54:37 | <davean> | Its the starting condition |
| 2021-07-01 20:54:51 | <tomsmeding> | my mental model (and I think dminuoso's) is that you're continuously feeding input at the tail end, and taking (and putting back of you over-took) at the head end |
| 2021-07-01 20:54:59 | <dminuoso> | Yes. |
| 2021-07-01 20:55:02 | <tomsmeding> | that's a queue with an additional "putback" operation |
| 2021-07-01 20:55:13 | <davean> | Theres no way to actually feed |
| 2021-07-01 20:55:16 | <dminuoso> | I somehow fail to understand davean's model. |
| 2021-07-01 20:55:19 | <davean> | you can't control the head of a streamign library |
| 2021-07-01 20:55:21 | <tomsmeding> | or, I guess, a stack with an additional shove-below-bottom operation |
| 2021-07-01 20:55:32 | <davean> | You don't get any queue operations in any streaming library I know of |
| 2021-07-01 20:55:34 | → | MQ-17J joins (~MQ-17J@d14-69-206-129.try.wideopenwest.com) |
| 2021-07-01 20:55:39 | <tomsmeding> | dminuoso: create lazy list of all data, then take and putback at the front |
| 2021-07-01 20:55:42 | <tomsmeding> | that take and putback is a stack |
All times are in UTC.