This is a post-hoc analysis tool for Apache Tez which splits an aggregated yarn log file to separate files into a hierarchical folder structure.
. ├── vc0525.your.domain.com_8041 │ └── container_e10_1575565459633_0004_01_000001 │ ├── container-localizer-syslog │ ├── dag_1575565459633_0004_1-tez-dag.pb.txt │ ├── dag_1575565459633_0004_1.dot │ ├── prelaunch.err │ ├── prelaunch.out │ ├── stderr │ ├── stdout │ ├── syslog │ ├── syslog_dag_1575565459633_0004_1 │ └── syslog_dag_1575565459633_0004_1_post ├── vc0526.your.domain.com_8041 │ └── container_e10_1575565459633_0004_01_000004 │ ├── container-localizer-syslog │ ├── prelaunch.err │ ├── prelaunch.out │ ├── stderr │ ├── stdout │ ├── syslog │ └── syslog_attempt_1575565459633_0004_1_00_000000_2 ├── vc0528.your.domain.com_8041 │ └── container_e10_1575565459633_0004_01_000002 │ ├── container-localizer-syslog │ ├── prelaunch.err │ ├── prelaunch.out │ ├── stderr │ ├── stdout │ ├── syslog │ └── syslog_attempt_1575565459633_0004_1_00_000000_0 ├── vc0529.your.domain.com_8041 │ └── container_e10_1575565459633_0004_01_000005 │ ├── container-localizer-syslog │ ├── prelaunch.err │ ├── prelaunch.out │ ├── stderr │ ├── stdout │ ├── syslog │ └── syslog_attempt_1575565459633_0004_1_00_000000_3 └── vc0536.your.domain.com_8041 └── container_e10_1575565459633_0004_01_000003 ├── container-localizer-syslog ├── prelaunch.err ├── prelaunch.out ├── stderr ├── stdout ├── syslog └── syslog_attempt_1575565459633_0004_1_00_000000_1
To use the tool, run e.g.
tez-log-splitter.sh application_1576254620247_0010
(app log is fetched from yarn)tez-log-splitter.sh ~/path/to/application_1576254620247_0010.log
(...when app log is already on your computer)tez-log-splitter.sh ~/path/to/application_1576254620247_0010.log.gz
(...when app log is already on your computer in gz)