| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <1439162006-11421-1-git-send-email-matthias.tafelmeier@gmx.net>
Date: Mon, 10 Aug 2015 01:13:16 +0200
From: Matthias Tafelmeier <matthias.tafelmeier@....net>
To: netdev@...r.kernel.org
Cc: hagen@...u.net, shemminger@...l.org, fw@...len.de,
edumazet@...gle.com, daniel@...earbox.net
Subject: full ss json support and general output simplification
TLDR:
- add full JSON support for ss
- Patchset provides a general and easy to use abstraction to extend ss later
- Patchset size is large to minimize daily use ("user" should not deal with
formation (json, human readble) later on)
- Patches 8/10 and 9/10 illustrate how to extend ss for new data to support human readble and json
output.
- Example_Usages: 1. ss -jt to print out all tcp related information formatted in json
2. ss --json -a to print out all info (also summary)
STATS:
Matthias Tafelmeier (10):
ss: rooted out ss type declarations for output formatters
ss: created formatters for json and hr
ss: removed obsolet fmt functions
ss: prepare timer for output handler usage
ss: framed skeleton for json output in ss
ss: replaced old output mechanisms with fmt handlers interfaces
ss: renaming and export of current_filter
ss: symmetrical subhandler output extension example
ss: symmetrical formatter extension example
ss: fixed free on local array for valid json output
misc/Makefile | 2 +-
misc/ss.c | 1010 +++++++++++++++++++---------------------------------
misc/ss_hr_fmt.c | 321 +++++++++++++++++
misc/ss_hr_fmt.h | 9 +
misc/ss_json_fmt.c | 466 ++++++++++++++++++++++++
misc/ss_json_fmt.h | 24 ++
misc/ss_out_fmt.c | 137 +++++++
misc/ss_out_fmt.h | 92 +++++
misc/ss_types.h | 186 ++++++++++
9 files changed, 1595 insertions(+), 652 deletions(-)
create mode 100644 misc/ss_hr_fmt.c
create mode 100644 misc/ss_hr_fmt.h
create mode 100644 misc/ss_json_fmt.c
create mode 100644 misc/ss_json_fmt.h
create mode 100644 misc/ss_out_fmt.c
create mode 100644 misc/ss_out_fmt.h
create mode 100644 misc/ss_types.h
--
Abstract:
This patch set originates from the necessity to upgrade ss with the possibility
to output in json format. Not to clutter up ss too much, the author of the
patch decided to come up with a simple distributor to handler approach. That
is, the distributor poses the mechanical interface which passes the output
requests coming from ss to the appropriate handler. This simplifies the
interaction with ss and provides a maximum of future extensiblity. Not to
forget, ss loses weight thereby since output implemented in ss itself does
migrate to the appropriate handler. Additionally, because types are shared
amongst handlers, the distributor and ss, the author conceived, that a separate
containter module for types has to be formed. In future, all type declarations
and extensins go there.
In sum, the patchset has this voluminous extent since there is no viable way
for putting out syntactically correct human readble and json in a simpler manner.
The requirement for convenient extensibility of output and data is
another justification for the patchset size.
Concept sketch:
formatter1
************
* *
* *
ss ~~~~~~~>zzzzzzz *
****************** ~ * *
* * ~ ###>fffffff *
* * ~ # * *
* * distributor~ # ************
* -------- * ********* ~ #
* - -------------- * * ~ #
* -------- * - * * ~ #
* * ---->++++ ~~~~ #
* * * * ~ # formatter2
* * ---->==== ######## ************
* -------- * - * * ~ # * *
* - -------------- * * ~ # * *
* -------- * ********* ~ # * *
* * ~~~~#~~>zzzzzzz *
* * # * *
* * ###>fffffff *
****************** * *
************
At the moment, the distributor is the ss_out_fmt module while two handlers are
up: namely the ss_json_fmt and the ss_hr_fmt (human readable). You can use
those modules as the main reference for own extensions.
Future Extension:
In the following, I will expand on the expandability of the formatter model.
The explanations advances from the minimal to the most sweeping extension in
mind.
Sub Format Handler Output
Sketch
FormatterX
***********************************
* *
* handlerX *
* °°°°°°°°°°°°°°°°°°°°°°°°°° *
* ° ° *
* ° xxxxxxxxxxxxxxc<.. ° *
* ° . ° *
* ° xxxxxxxxxxxxxxc<.. potential context
* ° new: . ° *
* ° +++++++++++++++... < * * * * * * * * * * *
* ° ° * *
* °°°°°°°°°°°°°°°°°°°°°°°°°° * *
* . * *
* . * *
* . * *
* handlerY * *
* °°°°°°°°°°°°°°°°°°°°°°°°°° * *
* ° ° * *
* ° xxxxxxxxxxxxxxx ° * *
* ° ° * *
* ° xxxxxxxxxxxxxxx ° * *
* ° ° * * expand symmetrically
* ° ° * *
* °°°°°°°°°°°°°°°°°°°°°°°°°° * *
* * *
* * *
* * *
*********************************** *
*
. *
. *
. *
*
FormatterY *
*********************************** *
* * *
* handlerX * *
* °°°°°°°°°°°°°°°°°°°°°°°°°° * *
* ° ° * *
* ° zzzzzzzzzzzzzzc<.. ° * *
* ° . ° * *
* ° zzzzzzzzzzzzzzc<.. potential context *
* ° new: . ° * *
* ° +++++++++++++++... < * * * * * * * * * * *
* ° ° *
* °°°°°°°°°°°°°°°°°°°°°°°°°° *
* . *
* . *
* . *
* handlerY *
* °°°°°°°°°°°°°°°°°°°°°°°°°° *
* ° ° *
* ° zzzzzzzzzzzzzzz ° *
* ° ° *
* ° zzzzzzzzzzzzzzz ° *
* ° ° *
* ° ° *
* °°°°°°°°°°°°°°°°°°°°°°°°°° *
* *
* *
***********************************
Explanation:
If you plan to expand a sub out handler function of a formatter, it
essentially boils down to adding a new printf with an according format and
probably a necessary predicate (condition). Nontheless, care must be taken not
to lose possible context interdependecies out of sight. An examble for the
latter would be the interdependecy of json coma setting terms – in compound
types, you do need a coma between consecutive elements.
More important is the issue about symmetric extensions. Except for the
tcp_out_fmt function implementation – where a macro (CHECK_FMT_ADAPT) is in
place to check for adaptions in the basic tcpstat data structure statically –
no general programmatic approach is in place yet which would prevent asymmetric
extensions. Up to someone devices a holistic solution, this patch relies on the
extenders to deal with asymmetries. Is the aim to have a new output feature
available in all semantically related handlers of n different formatters, then
the expander has to adapt n handlers as shown in the sketch above.
Example:
Let's take a look at the ss_json_fmt formatter module for a concrete example.
Were you to to extend the formatting handler tcp_stats_json_fmt, then it could
look as follows:
*** PSEUDO_CODE ***
static void tcp_stats_json_fmt(struct tcpstat *s)
{
char b1[64];
char indent1[] = "\t";
char indent2[] = "\t\t";
[...]
if (s->has_ts_opt) {
printf(",\n%s\"ts\": \"true\"", indent1);
}
if (s->has_sack_opt) {
printf(",\n%s\"sack\": \"true\"", indent1);
}
--->>> if (s->new_info) {
printf(",\n%s\"new_info\": \"XYZ\"", indent1);
}
[...]
}
and for SYMMETRY reasons extend on hr side as well:
static void tcp_stats_hr_fmt(struct tcpstat *s)
{
[...]
if (s->has_ts_opt)
printf(" ts");
if (s->has_sack_opt)
printf(" sack");
--->>> if (s->new_info)
printf(" *content of new info*");
[...]
}
Extend with further Format Handler in Formatter
Sketch:
-provides symmetrical extension intendet
formatter1
/--------------------------\
| |
| spec_handlerX |
| ################ <** |
| ################ * |
distributor | . * |
O-------------------------O | . * |
| centr_hub | | new_spec_handler * |
| ***>01111110--------------------------+ | ++++++++++++++++<~ * |
| * 02222220----------------------+ | | ++++++++++++++++ ~ * |
| * | | | | ~ * |
| * | | | | handler_hub1 ~ * |
| * _______________ | | +--------->0#########0****~** |
| * |gen_handlerX | | | | 0+++++++++0~~~~~ |
| ********** | | | | |
| * |_______________| | | | |
| * . | | \--------------------------/
| * . | | .
| * . | | .
| * +++++++++++++++++ | | formatterN .
| ***new_gen_handler+ | | /--------------------------\
| + + | | | |
| +++++++++++++++++ | | | spec_handlerX |
| | | | ################ <** |
O-------------------------O | | ################ * |
| | . * |
| | . * |
| | new_spec_handler * |
| | ++++++++++++++++<~ * |
| | ++++++++++++++++ ~ * |
| | ~ * |
| | handler_hubN ~ * |
+------------->0#########0****~** |
| 0+++++++++0~~~~~ |
| |
| |
\--------------------------/
Explanation:
As the sketch shows, the distributor works with the help of virtual function
pointer in order to act as a call flow switch. It switches to the approriate
formatter module and its handlers depending on the chosen output format by ss
command input.
So, to add a new formatter handler symmetrically (up to now that is the only
sensibly conceivable case), the extender must implement a new generic handler
in the distributor and the specific handlers in the formatters. Then the hub
vtable structure type has to be broadend to contain the new function pointer
type for the generic handler. After that, he has to extend and update all
handler hubs with the new handlers location information (function pointer). The
latter ensures the generic switching mechanism used by the generic handler
keeps to be upheld.
Example:
Let's say we want the new "foo" data for every output format retrievable via
ss. Up to now, we have the ss_out_fmt module as the distributor and two
specific handlers: for one the ss_hr_fmt and secondly the ss_json_fmt module.
So we need a specific handler implementation in ss_hr_fmt and ss_json_fmt
modules and after that update the corresponding vtables (handler_hubs) in the
modules. After that, the distributor, namely ss_out_fmt module, has to get a
generic handler that switches via its vtable hub to either the json formatter
or the human readable formatter, depending on what fmt_type has been chosen by ss.
Before the vtables in the specific modules can be updated, struct fmt_op_hub
which is found in ss_out_fmt's interface header has get extendend with the new
function pointer type.
As soon the new generic handler has been exported via the ss_out_fmt.h module
interface, ss can use the new fmt handler to print out info. It can simply call
the generic function and does not have to deal with formatting specific issues.
In the following pseudo code, the additions are marked with an arrow.
*** PSEUDO_CODE ***
### ss_out_fmt.h ###
struct fmt_op_hub {
void (*tcp_stats_fmt)(struct tcpstat *s);
void (*tcp_timer_fmt)(struct tcpstat *s);
[...]
void (*packet_details_fmt)(struct packet_diag_info * pinfo,
struct packet_diag_ring * ring_rx,
struct packet_diag_ring * ring_tx,
uint32_t fanout,
bool has_fanout);
void (*packet_show_ring_fmt)(struct packet_diag_ring *ring);
->>> void (*new_gen_foo_handler_fmt)(int xy);
};
### ss_out_fmt.c ###
[...]
void sock_users_fmt(char *out)
{
fmt_op_hub[fmt_type]->sock_users_fmt(out);
}
->>> void new_gen_foo_handler_fmt(int xy)
{
fmt_op_hub[fmt_type]->spec_foo_handler_fmt(out);
}
[...]
### ss_hr_fmt.c ###
[...]
static void sock_users_hr_fmt(char *out)
{
printf(" users:(%s)", out);
}
->>> void spec_foo_handler_hr_fmt(int xy)
{
printf(" foo: %d", xy);
}
[...]
const struct fmt_op_hub hr_output_op = {
.tcp_stats_fmt = tcp_stats_hr_fmt,
.tcp_timer_fmt = tcp_timer_hr_fmt,
[...]
.packet_details_fmt = packet_details_hr_fmt,
.packet_show_ring_fmt = packet_show_ring_hr_fmt,
->>> .new_gen_foo_handler_fmt = spec_foo_handler_hr_fmt
};
### ss_json_fmt.c ###
[...]
static void sock_users_json_fmt(char *out)
{
make_userout_valid(out);
printf(",\n\t\"users\": \"%s\"", out);
}
->>> void spec_foo_handler_json_fmt(int xy)
{
printf(",\n\t\"foo\": \"%d\"", xy);
}
[...]
const struct fmt_op_hub json_output_op = {
.tcp_stats_fmt = tcp_stats_json_fmt,
.tcp_timer_fmt = tcp_timer_json_fmt,
[...]
.packet_details_fmt = packet_details_json_fmt,
.packet_show_ring_fmt = packet_show_ring_json_fmt,
->>> .new_gen_foo_handler_fmt = spec_foo_handler_json_fmt
};
Extend for another Formatter
Sketch:
The Sketch for handler extension should be sufficient for conveying the concept.
Just think of another formatter after formatterN and a new entry in the central
vtable of the distributor to reach this new formatter.
Explanation:
Nothing breathtaking has to be done when someone needs an new formatter module
for let's be image – out of pure hypothetical endeavors – xml ss output. First,
implement the new formatter with all the offered interfaces in the
distributor. Register all handlers in the local specific vtable hub.
Then, register the local vtable hub in the generic vtable hub of the
distributor to reach your new handler when chosen. Provide the client
code - here ss - with a new fmt_type option acceptance. Before the
option can do anything, you have to declare the new fmt_type. That's
it. No further adaptions in ss would be necessary.
Example:
Extensions are highlighted with an arrow, as previously done.
### ss_out_fmt.h ###
enum out_fmt_type { FMT_HR, FMT_JSON,->>> FMT_NEW};
### ss_out_fmt.c ###
[...]
const struct fmt_op_hub *fmt_op_hub[] = {
/*human readble */
&hr_output_op,
/*json */
&json_output_op,
/*new */
&new_output_op
};
[...]
### ss_new_fmt.c ###
[...]
->>> static void sock_users_new_fmt(char *out)
{
make_userout_valid(out);
printf("NEW OUTFMT: %s", out);
}
[...]
const struct fmt_op_hub new_output_op = {
.tcp_stats_fmt = ->>> tcp_stats_new_fmt,
.tcp_timer_fmt = ->>> tcp_timer_new_fmt,
[...]
.sock_users_fmt = ->>> sock_users_new_fmt,
[...]
.packet_details_fmt = ->>> packet_details_new_fmt,
.packet_show_ring_fmt =->>> packet_show_ring_new_fmt,
};
### ss.c ###
[...]
int json_output = 0;
->>> int new_output = 0;
[...]
case 'j':
fmt_type = FMT_JSON;
json_output = 1;
break;
->>>case 'N':
->>> fmt_type = FMT_NEW;
->>> new_output = 1;
->>> break;
case 'h':
[...]
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists