We had a flake of form:
```
2025-11-18T04:42:23.489Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: wake delay for WIRE_CHANNEL_REESTABLISH: 6789msec
```
Which happened as we're shutting down. Some investigation revealed
the cause: `dev-memleak` can be extremely slow. Fair enough.
So we change `dev-memleak` to call connectd first, and connectd uses
that as a trigger to stop complaining about delays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You can now simply add per-tal-object helpers for memleak, but our older pattern required
calling memleak functions explicitly during memleak handling. Hash tables in particular need
to be dynamically allocated (we override the allocators using htable_set_allocator and assume
this), so it makes sense to have a helper macro that does all three.
This eliminates a huge amount of code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We ask each channeld to report its leaks, and keep a refcount of how
many are outstanding. When the channeld replies, or dies, we decrement
the count in a destructor.
But if the last channeld we're waiting for dies, we can call the
destructor in an unexpected place, and thus run leak detection when
we were partway through some operation, which gives a false positive.
Here's a backtrace showing it:
```
0x5f93405897e3 send_backtrace
common/daemon.c:33
0x5f93405381cf finish_report
lightningd/memdump.c:139
0x5f93405382a0 leak_detect_req_done
lightningd/memdump.c:172
0x5f9340705c41 notify
ccan/ccan/tal/tal.c:243
0x5f9340705ca5 del_tree
ccan/ccan/tal/tal.c:437
0x5f9340705ce8 del_tree
ccan/ccan/tal/tal.c:447
0x5f93407061f3 tal_free
ccan/ccan/tal/tal.c:532
0x5f9340563f5a subd_release_channel
lightningd/subd.c:924
0x5f934050fb91 channel_set_owner
lightningd/channel.c:31
0x5f9340511b84 channel_err
lightningd/channel.c:1081
0x5f93405121a3 channel_fail_transient
lightningd/channel.c:1095
0x5f934054e547 peer_channels_cleanup
lightningd/peer_control.c:187
0x5f9340550411 peer_connected
lightningd/peer_control.c:1689
0x5f9340525101 connectd_msg
lightningd/connect_control.c:629
```
Instead, do the lightningd detection up-front, where it's in a
clean environment.
Reported-by: Shahana Farooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes `check` much more thorough, and useful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` now does much more checking on every command (not just basic parameter types).
This makes it easier to use outside simple subds, and now lightningd can
simply dump to log rather than returning JSON.
JSON formatting was a lot of work, and we only did it for lightningd, not for
subdaemons. Easier to use the logs in all cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And require --developer to use them.
Also refuse redirection to deprecated APIs if deprecated APIs are disabled!
Changelog-Removed: `dev-sendcustommsg` (use `sendcustommsg`, which was added in v0.10.1)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After connecting 100,000 peers with one channel each (not all at
once!), we see various places where we exhibit O(N^2) behaviour.
Fix these by keeping a map of id->peer instead of a simple
linked-list.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Add memleak_ignore_children() so callers can do exclusions themselves.
Having two exclusions was always such a hack!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have them split over common/param.c, common/json.c,
common/json_helpers.c, common/json_tok.c and common/json_stream.c.
Change that to:
* common/json_parse (all the json_to_xxx routines)
* common/json_parse_simple (simplest the json parsing routines, for cli too)
* common/json_stream (all the json_add_xxx routines)
* common/json_param (all the param and param_xxx routines)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of doing this weird chaining, just call them all at once and
use a reference counter.
To make it simpler, we return the subd_req so we can hang a destructor
off it which decrements after the request is complete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).
config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before:
Ten builds, laptop -j5, no ccache:
```
real 0m36.686000-38.956000(38.608+/-0.65)s
user 2m32.864000-42.253000(40.7545+/-2.7)s
sys 0m16.618000-18.316000(17.8531+/-0.48)s
```
Ten builds, laptop -j5, ccache (warm):
```
real 0m8.212000-8.577000(8.39989+/-0.13)s
user 0m12.731000-13.212000(12.9751+/-0.17)s
sys 0m3.697000-3.902000(3.83722+/-0.064)s
```
After:
Ten builds, laptop -j5, no ccache: 8% faster
```
real 0m33.802000-35.773000(35.468+/-0.54)s
user 2m19.073000-27.754000(26.2542+/-2.3)s
sys 0m15.784000-17.173000(16.7165+/-0.37)s
```
Ten builds, laptop -j5, ccache (warm): 1% faster
```
real 0m8.200000-8.485000(8.30138+/-0.097)s
user 0m12.485000-13.100000(12.7344+/-0.19)s
sys 0m3.702000-3.889000(3.78787+/-0.056)s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Rename memleak_enter_allocations to memleak_find_allocations.
2. Unify scanning for pointers into memleak_remove_region / memleak_remove_pointer.
3. Document the functions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that other directories were explicitly depending on the generated
file, instead of relying on their (already existing) dependency on
$(LIGHTNINGD_HSM_CLIENT_OBJS), so we remove that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than reaching into data structures, let them register their own
callbacks. This avoids us having to expose "memleak_remove_xxx"
functions, and call them manually.
Under the hood, this is done by having a specially-named tal child of
the thing we want to assist, containing the callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we are walking the entire allocation tree anyway, and access the tal
metadata anyway, we can just as well also track the size of the memory
allocations to simplify debugging of memory use.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Direct leak of 1024 byte(s) in 2 object(s) allocated from:
#0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
#1 0x55d11b782c96 in timer_default_alloc ccan/ccan/timer/timer.c:16
#2 0x55d11b7832b7 in add_level ccan/ccan/timer/timer.c:166
#3 0x55d11b783864 in timer_fast_forward ccan/ccan/timer/timer.c:334
#4 0x55d11b78396a in timers_expire ccan/ccan/timer/timer.c:359
#5 0x55d11b774993 in io_loop ccan/ccan/io/poll.c:395
#6 0x55d11b72322f in plugins_init lightningd/plugin.c:1013
#7 0x55d11b7060ea in main lightningd/lightningd.c:664
#8 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
To fix this, we actually make 'ld->timers' a pointer, so we can clean
it up last of all. We can't free it before ld, because that causes
timers to be destroyed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
"result" should always be an object (so that we can add new fields),
so make that implicit in json_stream_success.
This makes our primitives well-formed: we previously used NULL as our
fieldname when calling the first json_object_start, which is a hack
since we're actually in an object and the fieldname is 'result' (which
was already written by json_object_start).
There were only two cases which didn't do this:
1. dev-memdump returned an array. No API guarantees on this.
2. shutdown returned a string.
I temporarily made shutdown return an empty object, which shouldn't
break anything, but I want to fix that later anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A new string field is added to the command structure and is specified at the creation of each native command, and in the JSON created by 'json_add_help_command()'.
We store it in a strmap. This means we call the jsonrpc handler earlier,
so all callers need to call param() before they do anything else; only
json_listaddrs and json_help needed fixing.
Plugins still use '[usage]' for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
memdump iterates through the various daemons asking them to check for
leaks.
We currently call openingds (there might be none), channelds (there
might be none), then hsmd synchronously (the other daemons). If hsmd
reports a leak, we'll fail the dev-memleak command immediately.
Change the order to call connectd first; that's always async, so we
can happily mark the command still pending.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This causes a compiler warning if we don't do something with the
result (hopefully return immediately!).
We use was_pending() to ignore the result in the case where we
complete a command in a callback (thus really do want to ignore
the result).
This actually fixes one bug: we didn't return after command_fail
in json_getroute with a bad seed value.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Usually, this means they return 'command_param_failed()' if param()
fails, and changing 'command_success(); return;' to 'return
command_success()'.
Occasionally, it's more complex: there's a command_its_complicated()
for the case where we can't exactly determine what the status is,
but it should be considered a last resort.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
json_escaped.[ch], param.[ch] and jsonrpc_errors.h move from lightningd/
to common/. Tests moved too.
We add a new 'common/json_tok.[ch]' for the common parameter parsing
routines which a plugin might want, taking them out of
lightningd/json.c (which now only contains the lightningd-specific
ones).
The rest is mainly fixing up includes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This (will) avoid the plugin having to walk back from the params object
as it currently does.
No code changes; I removed UNUSED and UNNEEDED labels from the other
parameters though (as *every* json_rpc callback needs to call param()
these days, they're *always* used).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
