Sometime last year I built a process exporter for macOS and needed per process network statistics.
The problem is defined as; for each process, how many bytes has it sent and received?
I knew nettop already provided this information
process name + pid -> bytes in / bytes out
So the question is, how does nettop do it?
In this article I try to reconstruct that
Basically inspect the binary, list the linked private framework, infer a few function signatures, segfault a few times, correct the ABI, then aggregate CoreFoundation dictionaries.
Starting with nettop
The first thing to do was confirm whether nettop was using some private framework directly.
file /usr/bin/nettop
On my machine:
/usr/bin/nettop: Mach-O universal binary with 2 architectures: [x86_64] [arm64e]
Then:
otool -L /usr/bin/nettop
The relevant line:
/System/Library/PrivateFrameworks/NetworkStatistics.framework/Versions/A/NetworkStatistics
So nettop links directly against NetworkStatistics.framework, a private framework.
The next step was to see which symbols it imports.
dyld_info -imports /usr/bin/nettop
The interesting set was:
NStatManagerCreate
NStatManagerAddAllRoutesWithFilter
NStatManagerAddAllTCPWithFilter
NStatManagerAddAllUDPWithFilter
NStatManagerQueryAllSourcesUpdate
NStatManagerSetFlags
NStatManagerSetInterfaceTraceFD
NStatSourceCopyProperties
NStatSourceCopyProperty
NStatSourceQueryDescription
NStatSourceRemove
NStatSourceSetCountsBlock
NStatSourceSetDescriptionBlock
NStatSourceSetRemovedBlock
There were also a lot of CoreFoundation keys:
kNStatSrcKeyPID
kNStatSrcKeyProcessName
kNStatSrcKeyRxBytes
kNStatSrcKeyRxPackets
kNStatSrcKeyTxBytes
kNStatSrcKeyTxPackets
kNStatSrcKeyInterface
kNStatSrcKeyLocal
kNStatSrcKeyRemote
kNStatSrcKeyProvider
That already gives the shape of the API:
- Create a manager.
- Subscribe to TCP and UDP sources.
- For each source, install callbacks for description, counts, and removal.
- Periodically query updates.
- Read process and counter fields from dictionaries.
At this point, we just need to find out the function signatures
dyld shared cache
I tried to inspect the framework binary directly:
nm -gjU /System/Library/PrivateFrameworks/NetworkStatistics.framework/NetworkStatistics
But on this macOS install the framework path is only a symlink:
NetworkStatistics -> Versions/Current/NetworkStatistics
and the actual Mach-O file is not there.
This is normal now. A lot of system frameworks live in the dyld shared cache instead of existing as normal files at their apparent paths. That makes the first pass slightly annoying because otool -L /usr/bin/nettop can show the dependency, but direct nm on the framework path fails.
dyld_info can still ask the shared cache for exports:
dyld_info -exports -all_dyld_cache | grep -E "NetworkStatistics|_NStat|_kNStat"
That revealed a few more useful functions beyond what nettop imports:
NStatManagerAddAllTCP
NStatManagerAddAllUDP
NStatManagerDestroy
NStatSourceCopyCounts
NStatSourceIdentifier
NStatSourceQueryCounts
NStatSourceQueryUpdate
The non-filter variants looked useful for a small prototype. I do not need filtering to prove per-process counters work, I can subscribe to all TCP and UDP sources and aggregate them myself.
Xcode’s full macOS SDK also had a private .tbd stub for the framework:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/PrivateFrameworks/NetworkStatistics.framework/Versions/A/NetworkStatistics.tbd
The Command Line Tools SDK on my machine did not.
I decided not to link with -framework NetworkStatistics. Instead I used dlopen and dlsym. That avoids depending on which SDK happens to be selected.
Find the block types
While we can make sense of the API surface from the imported names, it’s a little tricky for block signatures. A quick strings pass over nettop helped:
strings -a /usr/bin/nettop
The useful strings were:
v16@?0^{__NStatSource=}8
v16@?0^{__CFDictionary=}8
v8@?0
These are Objective-C block type encodings.
Interpretation:
v16@?0^{__NStatSource=}8is a block returningvoidand taking anNStatSource *.v16@?0^{__CFDictionary=}8is a block returningvoidand taking aCFDictionary *.v8@?0is a no-argument block returningvoid.
So I started with these opaque types:
typedef struct __NStatManager *NStatManagerRef;
typedef struct __NStatSource *NStatSourceRef;
typedef void (^NStatSourceBlock)(NStatSourceRef source);
typedef void (^NStatDictionaryBlock)(CFDictionaryRef dictionary);
And then a guessed API:
NStatManagerRef NStatManagerCreate(CFAllocatorRef allocator,
dispatch_queue_t queue,
NStatSourceBlock block);
void NStatManagerDestroy(NStatManagerRef manager);
void NStatManagerAddAllTCP(NStatManagerRef manager);
void NStatManagerAddAllUDP(NStatManagerRef manager);
void NStatManagerQueryAllSourcesUpdate(NStatManagerRef manager,
dispatch_block_t completion);
void NStatSourceQueryDescription(NStatSourceRef source);
void NStatSourceSetDescriptionBlock(NStatSourceRef source,
NStatDictionaryBlock block);
void NStatSourceSetCountsBlock(NStatSourceRef source,
NStatDictionaryBlock block);
void NStatSourceSetRemovedBlock(NStatSourceRef source,
NStatSourceBlock block);
The code above is the corrected version
Segfault bitch
My first guess for NStatManagerCreate was:
NStatManagerRef NStatManagerCreate(dispatch_queue_t queue,
NStatSourceBlock block);
That compiled and then crashed immediately.
Disassembly here we come. On arm64, function arguments are passed in registers x0, x1, x2, etc. So looking at what nettop puts in those registers before calling NStatManagerCreate tells us the real signature.
dyld_info -arch arm64e -disassemble /usr/bin/nettop
Around the call site:
x0 = kCFAllocatorDefault
x1 = __dispatch_main_q
x2 = block pointer
bl _NStatManagerCreate
So the actual signature was:
NStatManagerRef NStatManagerCreate(CFAllocatorRef allocator,
dispatch_queue_t queue,
NStatSourceBlock block);
That fixed the crash.
Then the manager was created, but my update call was still wrong. I had guessed:
void NStatManagerQueryAllSourcesUpdate(NStatManagerRef manager);
nettop showed a second argument:
x0 = manager
x1 = block pointer
bl _NStatManagerQueryAllSourcesUpdate
That lines up with the v8@?0 block encoding, so the corrected version is:
void NStatManagerQueryAllSourcesUpdate(NStatManagerRef manager,
dispatch_block_t completion);
Load the framework
The dynamic loader part is straightforward.
const char *paths[] = {
"/System/Library/PrivateFrameworks/NetworkStatistics.framework/Versions/A/NetworkStatistics",
"/System/Library/PrivateFrameworks/NetworkStatistics.framework/NetworkStatistics",
NULL,
};
for (size_t i = 0; paths[i] != NULL; i++) {
api->handle = dlopen(paths[i], RTLD_LAZY | RTLD_LOCAL);
if (api->handle != NULL) {
break;
}
}
Then resolve functions:
api->manager_create = dlsym(handle, "NStatManagerCreate");
api->manager_add_all_tcp = dlsym(handle, "NStatManagerAddAllTCP");
api->manager_add_all_udp = dlsym(handle, "NStatManagerAddAllUDP");
api->manager_query_all_sources_update =
dlsym(handle, "NStatManagerQueryAllSourcesUpdate");
The keys are exported as global CFStringRef values, so you resolve the address of the symbol and then dereference it:
static CFStringRef load_cfstring_symbol(void *handle, const char *name)
{
CFStringRef *slot = (CFStringRef *)dlsym(handle, name);
if (slot == NULL) {
return NULL;
}
return *slot;
}
Then:
api->key_pid = load_cfstring_symbol(handle, "kNStatSrcKeyPID");
api->key_process_name = load_cfstring_symbol(handle, "kNStatSrcKeyProcessName");
api->key_rx_bytes = load_cfstring_symbol(handle, "kNStatSrcKeyRxBytes");
api->key_tx_bytes = load_cfstring_symbol(handle, "kNStatSrcKeyTxBytes");
api->key_rx_packets = load_cfstring_symbol(handle, "kNStatSrcKeyRxPackets");
api->key_tx_packets = load_cfstring_symbol(handle, "kNStatSrcKeyTxPackets");
Subscribe to sources
The manager callback receives an NStatSourceRef whenever the framework discovers a source. A source here is a network statistics object, roughly a TCP or UDP flow/socket-like thing depending on provider.
For each source I install three blocks:
static void source_added(NStatSourceRef source)
{
upsert_source(source);
g_api.source_set_description_block(source, ^(CFDictionaryRef dictionary) {
update_description(source, dictionary);
});
g_api.source_set_counts_block(source, ^(CFDictionaryRef dictionary) {
update_counts(source, dictionary);
});
g_api.source_set_removed_block(source, ^(NStatSourceRef removed_source) {
struct source_state *state = find_source(removed_source);
if (state != NULL) {
state->active = false;
}
});
g_api.source_query_description(source);
}
The description dictionary is where I pull attribution:
static void update_description(NStatSourceRef source, CFDictionaryRef dictionary)
{
struct source_state *state = upsert_source(source);
CFTypeRef pid_value = CFDictionaryGetValue(dictionary, g_api.key_pid);
cf_number_to_pid(pid_value, &state->pid);
CFTypeRef process_value =
CFDictionaryGetValue(dictionary, g_api.key_process_name);
cf_string_to_c(process_value, state->process, sizeof(state->process));
}
The counts dictionary is where the traffic counters come from:
static void update_counts(NStatSourceRef source, CFDictionaryRef dictionary)
{
struct source_state *state = upsert_source(source);
cf_number_to_u64(CFDictionaryGetValue(dictionary, g_api.key_rx_bytes),
&state->rx_bytes);
cf_number_to_u64(CFDictionaryGetValue(dictionary, g_api.key_tx_bytes),
&state->tx_bytes);
cf_number_to_u64(CFDictionaryGetValue(dictionary, g_api.key_rx_packets),
&state->rx_packets);
cf_number_to_u64(CFDictionaryGetValue(dictionary, g_api.key_tx_packets),
&state->tx_packets);
}
Then setup is just:
NStatManagerRef manager =
NStatManagerCreate(kCFAllocatorDefault, queue, ^(NStatSourceRef source) {
source_added(source);
});
NStatManagerAddAllTCP(manager);
NStatManagerAddAllUDP(manager);
and periodic refresh:
NStatManagerQueryAllSourcesUpdate(manager, ^{
print_totals();
});
One small thing I copied from the nettop pattern: after installing blocks for a source, call NStatSourceQueryDescription(source), but let manager updates drive the counts callback.
Aggregating per process
NetworkStatistics gives source-level counters. nettop -P is a per-process summary, so we still need an aggregation layer.
The state I keep per source is:
struct source_state {
NStatSourceRef source;
pid_t pid;
char process[PROC_PIDPATHINFO_MAXSIZE];
uint64_t rx_bytes;
uint64_t tx_bytes;
uint64_t rx_packets;
uint64_t tx_packets;
bool active;
bool has_description;
bool has_counts;
};
On each print, walk all active sources, group by pid, and add up rx/tx bytes and packets.
for (size_t i = 0; i < g_source_count; i++) {
const struct source_state *source = &g_sources[i];
if (!source->active || !source->has_counts) {
continue;
}
struct process_totals *total =
upsert_total(&totals, &total_count, &total_capacity, source);
total->rx_bytes += source->rx_bytes;
total->tx_bytes += source->tx_bytes;
total->rx_packets += source->rx_packets;
total->tx_packets += source->tx_packets;
}
Then sort by total bytes.
This is not a full nettop replacement. It does not render connection rows, traffic class, TCP state, RTT, interface type, route mode, DNS names, etc. But for the thing I needed originally (per process network counters), this is the core of it.
Test
To validate it, I compared the output with stock nettop:
nettop -l 1 -P -n
Then ran the prototype:
./nstat-top -s 2 -i 1
Sample output:
SAMPLE 1: 329 tracked sources, 54 processes
PID PROCESS RX_BYTES TX_BYTES RX_PACKETS TX_PACKETS
644 mDNSResponder 461396756 224059294 1813546 614809
75399 CloudflareWARP 80926479 20760223 122954 83756
97550 Google Chrome Helper 92963716 2804013 226380 7139
Sanity check against nettop:
mDNSResponder: 440 MiB in, 213 MiB out
The prototype reported:
461396756 bytes in
224059294 bytes out
Convert them:
461396756 / 1024 / 1024 = 440.0 MiB
224059294 / 1024 / 1024 = 213.7 MiB
That was good enough for me. The counters line up with nettop’s.