AMBARI-23456. Add more markdown docs
diff --git a/ambari-logsearch-docs/pom.xml b/ambari-logsearch-docs/pom.xml
index eefdc6b..90a19a9 100644
--- a/ambari-logsearch-docs/pom.xml
+++ b/ambari-logsearch-docs/pom.xml
@@ -28,6 +28,7 @@
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<name>Ambari Logsearch Docs</name>
+ <description>Ambari Logsearch Docs</description>
<build>
<resources>
<resource>
diff --git a/docs/about.md b/docs/about.md
index 4fb1258..11625d6 100644
--- a/docs/about.md
+++ b/docs/about.md
@@ -13,4 +13,14 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+## Contribution Guide
+
+https://cwiki.apache.org/confluence/display/AMBARI/How+to+Contribute
+
+(That is the ambari contribution guide, everything is the same here except use ambari-logsearch repository instead of ambari)
+
+## License
+
+[Apache 2.0 LICENSE](https://www.apache.org/licenses/LICENSE-2.0)
\ No newline at end of file
diff --git a/docs/add_new_input.md b/docs/add_new_input.md
index 4fb1258..24c3e30 100644
--- a/docs/add_new_input.md
+++ b/docs/add_new_input.md
@@ -13,4 +13,96 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+## Adding New Logs to Monitor
+
+#### Input / Filter (shipper) descriptors
+
+For adding a new log that will be monitored by Log Feeder, first, it is needed to understand how Log Feeder registers and use those logs as inputs.
+All the input log descriptors are stored in Log Feeder configuration directory with `input.config-<service>.json` naming pattern. (see: [Shipper configurations](shipper_configurations.md))
+
+Based on these descriptors, the Log Feeder will start to monitor the inputs that are defined inside the descriptor as inputs.
+After the configs are first processed, those will be handled/uploaded by a configuration handler, that should be an implementation of the Config API.
+
+#### Config API
+
+The Config API is responsible for to load and handle input (and filter) descriptors for Log Feeder. The idea is to store those configurations in a shared system
+and Log Feeders should be notified about any of the input configuration changes.
+It has different implementations, the default one is ZooKeeper (there is 2 other one right now: Solr and Local as well). Also with using/enabling the Config API,
+the user will be able to edit / test / add input descriptors.
+
+#### Support new input descriptor(s) for Ambari components
+
+- Create a new jinja template: `<SERVICE_LOCATION>/package/templates/input.config-<service_name>.json.j2`
+
+```jinja2
+{
+ "input":[
+ {
+ "type":"my_log",
+ "rowtype":"service",
+ "path":"{{default('/configurations/myservice-env/log_dir', '/var/log/myservice')}}/my_log*.log"
+ },
+ {
+ "type":"myother_log",
+ "rowtype":"service",
+ "path":"{{default('/configurations/myservice-env/log_dir', '/var/log/myservice')}}/myother_log*.log"
+ }
+ ],
+ "filter": [
+ ...
+ ]
+```
+
+- Change the stack code to write the input descriptor (add this to the start script):
+
+```python
+from resource_management.core.source import Template
+from resource_management.libraries.functions.default import default
+from resource_management.libraries.functions.generate_logfeeder_input_config import generate_logfeeder_input_config
+
+# ...
+ # that will write input.config.myservice.json file to /usr/lib/ambari-logsearch-logfeeder/conf folder
+ generate_logfeeder_input_config('myservice', Template("input.config-myservice.json.j2", extra_imports=[default]))
+# ...
+```
+- Edit the `metainfo.xml`: add logId mapping for components (logIds are log types in input descriptors)
+```xml
+<!-- ... -->
+ <components>
+ <component>
+ <name>MYCOMPONENT</name>
+ <logs>
+ <log>
+ <logId>my_log</logId>
+ <primary>true</primary>
+ </log>
+ <log>
+ <logId>myother_log</logId>
+ </log>
+ </logs>
+ <!-- ... -->
+ <component>
+ <components>
+ <!-- ... -->
+```
+From that point - during service start - the input config desriptor will be written into `/usr/lib/ambari-logsearch-logfeeder/conf`. The Log Feeder applications monitor the files in that folder, if there is a new one, it will start to monitor those inputs. Note that if the Config API is configured to use ZooKeeper, these configurations are only used from bootstrapping, after these files are uploaded to ZooKeeper, you will be onlyable to edit the input descriptos by Log Search Portal (or manually edit the znodes)
+
+#### Add / Edit / Test input desriptors by Log Search Portal
+
+On the Log Search Portal (if you go into `Configuration Editor`) you can edit (or add) existing input / filter configurations:
+
+
+
+Also you can add test samples that you can validate:
+
+
+
+If the validation is successful, you should got a map response with the created fields:
+
+
+
+### Add input descriptors without Ambari
+
+As you need the `input.config-<service>.json` files in `/usr/lib/ambari-logsearch-logfeeder/conf` folder (or where your configuration is located), it is enough to just create a new file at that location.
\ No newline at end of file
diff --git a/docs/cloud_mode.md b/docs/cloud_mode.md
index 4fb1258..731c6d2 100644
--- a/docs/cloud_mode.md
+++ b/docs/cloud_mode.md
@@ -13,4 +13,8 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+### Log Feeder: cloud mode
+
+TODO
\ No newline at end of file
diff --git a/docs/collections.md b/docs/collections.md
new file mode 100644
index 0000000..7031cad
--- /dev/null
+++ b/docs/collections.md
@@ -0,0 +1,62 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+### Collections
+
+By default there are 3 main storage layer abstractions in Log Search:
+
+- Service logs (default name: `hadoop_logs`)
+- Audit logs (default name: `audit_logs`)
+- Metadata (default name: `logsearch_metadata`)
+
+Service logs collection is responsible to store most of the logs (by type), except audit related data, for those use the audit logs collection.
+The metadata collection is used to store Log Search UI related (dynamic) configurations/settings.
+
+### Schema fields
+
+Minimal required service log fields in Solr to make it work with the UI:
+
+- id (string, unique - identifier for Solr doc)
+- log_message
+- type (string - log type)
+- logtime (timestamp)
+- seq_num (numeric - sequence number for log events, useful to not sort only by date)
+- level (string - log level for logs, e.g.: WARN)
+- host (string)
+- cluster (string)
+
+see more: [Service logs schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema)
+
+Minimal required audit log fields in Solr to make it work with the UI:
+
+- id (string, unique - identifier for Solr doc)
+- evtTime (timestamp)
+- repo (string - represents the audit source type)
+- seq_num (numeric - sequence number for log events, useful to not sort only by date)
+
+see more: [Audit logs schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/audit_logs/conf/managed-schema)
+
+Fields for metadata:
+- id (string, unique - identifier for Solr doc)
+- name (string - metadata identifier)
+- username (string - for identify user related data)
+- type (string - type of the metadata)
+- value (string - can be anything)
+
+### Customize field names on the Log Search UI
+
+Field name labels on the UI can be customized in `logsearch.properties`, see: [AMBARI-22842](https://issues.apache.org/jira/browse/AMBARI-22842)
\ No newline at end of file
diff --git a/docs/getting_started.md b/docs/development.md
similarity index 93%
rename from docs/getting_started.md
rename to docs/development.md
index f51f3ec..9a27bef 100644
--- a/docs/getting_started.md
+++ b/docs/development.md
@@ -15,8 +15,10 @@
limitations under the License.
-->
-## Installation
+### Backend development
-## Build
+TODO
-## Requirements
\ No newline at end of file
+### UI development
+
+TODO
\ No newline at end of file
diff --git a/docs/docs.md b/docs/docs.md
index 4fb1258..efd7ebe 100644
--- a/docs/docs.md
+++ b/docs/docs.md
@@ -13,4 +13,16 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+### Rest API doc
+
+Swagger specification: [logsearch-swagger.yaml](api-docs/logsearch-swagger.yaml)
+
+### Javadoc
+
+Generate javadoc with the following command from the project root:
+
+```bash
+make javadoc
+```
\ No newline at end of file
diff --git a/docs/images/add_service.png b/docs/images/add_service.png
new file mode 100644
index 0000000..fc97f04
--- /dev/null
+++ b/docs/images/add_service.png
Binary files differ
diff --git a/docs/images/add_service_wizard.png b/docs/images/add_service_wizard.png
new file mode 100644
index 0000000..cf83462
--- /dev/null
+++ b/docs/images/add_service_wizard.png
Binary files differ
diff --git a/docs/images/architecture_overview.jpg b/docs/images/architecture_overview.jpg
new file mode 100644
index 0000000..476e81f
--- /dev/null
+++ b/docs/images/architecture_overview.jpg
Binary files differ
diff --git a/docs/images/edit_configs.jpg b/docs/images/edit_configs.jpg
new file mode 100644
index 0000000..7599685
--- /dev/null
+++ b/docs/images/edit_configs.jpg
Binary files differ
diff --git a/docs/images/test_sample.jpg b/docs/images/test_sample.jpg
new file mode 100644
index 0000000..1f5e3f3
--- /dev/null
+++ b/docs/images/test_sample.jpg
Binary files differ
diff --git a/docs/images/test_sample_result.jpg b/docs/images/test_sample_result.jpg
new file mode 100644
index 0000000..ba208a9
--- /dev/null
+++ b/docs/images/test_sample_result.jpg
Binary files differ
diff --git a/docs/index.md b/docs/index.md
index 17fa30e..e994a61 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -18,4 +18,31 @@
Log aggregation, analysis, and visualization for Ambari managed (or any other) services.
-## Architecture
\ No newline at end of file
+## Features
+
+- Parse / aggregate and ship logs
+- Send and index logs in Solr
+- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS / ADLS / WASB)
+- Fultext Search in logs (if the logs are shipped to Solr)
+- JWT/SSO support
+- Support testing the log parsing on the UI
+
+## Architecture
+
+- Log Feeder: agent component on all hosts to monitor and shipping logs.
+- Log Search Portal: REST API + UI for rendering logs
+- Solr (Optional - default): storage for logs, used by both Log Search Portal and Log Feeder
+- ZooKeeper (Optional - default): configuration service for Solr, Log Search and Log Feeder
+- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder [cloud mode](cloud_mode.md)
+
+
+
+## Contents
+
+- [1, Installation](installation.md)
+- [2. Collections](collections.md)
+- [3. Adding new logs to monitor](add_new_input.md)
+- [4. Development guide](development.md)
+- [5. Using Log Feeder in Cloud mode](cloud_mode.md)
+- [6. Contribution Guide and License](about.md)
+
diff --git a/docs/installation.md b/docs/installation.md
new file mode 100644
index 0000000..05dd8d9
--- /dev/null
+++ b/docs/installation.md
@@ -0,0 +1,138 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+## Installation
+
+#### Requirements for building the project
+
+- Java 8+
+- Maven 3.5.x +
+- `rpm-build` on Mac OS X
+
+#### Checkout the project
+
+```bash
+git clone https://github.com/apache/ambari-logsearch
+cd ambari-logsearch
+```
+#### Build Log Search packages
+
+- a.) Create RPM packages (outputs: ambari-logsearch-assembly/target/rpms/*)
+```bash
+make rpm
+```
+
+- b.) Create DEB packages (outputs: ambari-logsearch-assembly/target/*.deb)
+```bash
+make deb
+```
+
+- c.) Create tarballs (outputs: ambari-logsearch-server/target/ambari-logsearch-portal.tar.gz and ambari-logsearch-logfeeder/target/ambari-logsearch-logfeeder.tar.gz)
+```bash
+make package
+```
+
+#### Install created packages
+
+- On RHEL / CentOS :
+```bash
+# Log Search Portal (server + UI)
+yum install -y /my/path/ambari-logsearch-portal.rpm
+# Log Feeder
+yum install -y /my/path/ambari-logsearch-logfeeder.rpm
+# or if a repo is available that contains logsearch packages
+yum install -y ambari-logsearch-portal
+yum install -y ambari-logsearch-logfeeder
+```
+- On Debian / Ubuntu :
+```bash
+# Log Search Portal (server + UI)
+dpkg -i /my/path/ambari-logsearch-portal.deb
+# Log Feeder
+dpkg -i /my/path/ambari-logsearch-logfeeder.deb
+# or if a repo is available that contains logsearch packages
+apt-get install ambari-logsearch-portal
+apt-get install ambari-logsearch-logfeeder
+```
+- With tarball :
+```bash
+mkdir logsearch
+tar -xzf ambari-logsearch-portal.tar.gz -C logsearch
+
+mkdir logfeeder
+tar -xzf ambari-logsearch-logfeeder.tar.gz -C logfeeder
+```
+
+#### Install by Ambari UI
+
+Click on Add Service page:
+
+
+
+Then select Log Search on Add Services page:
+
+
+
+Then (after setting the right configuration on Configure services page) you can start the Log Search / Log Feeder deployments.
+
+#### Component names to use in Ambari Blueprints:
+
+- Log Search Portal: `LOGSEARCH_SERVER` (1 node, or 1+, as Log Search Portal is stateless)
+- Log Feeder: `LOGSEARCH_LOGFEEDER`(All nodes)
+
+#### Requirements for using Log Search
+
+- `Solr` 6.x+ and `ZooKeeper` (as Solr is used in Cloud mode) is required for Log Search Portal - Ambari Infra Solr can be used as a built-in Solr service in Ambari, but of course it can use an external Solr as well.
+- Log Feeder in Cloud mode can work without Solr and ZooKeeper (that means logs will be shipped to HDFS/S3/GCS/WASB/ADLSv2 ... but Log Search Portal won't be able to display data from those Storages)
+
+#### Configurations
+
+##### Log Search Portal configuration:
+
+Required configuration files for Log Search Portal at `/usr/lib/ambari-logsearch-portal/conf`:
+
+- logsearch.properties
+- logsearch-env.sh
+- log4j2.yml
+- info.properties
+- default.properties
+
+##### Log Feeder configuration:
+
+Required configuration files for Log Feeder at `/usr/lib/ambari-logsearch-logfeeder/conf`:
+
+- logfeeder.properties
+- logfeeder-env.sh
+- log4j2.yml
+- alias_config.json
+- global.config.json
+- output.config.json
+- input.config-<service_name>.json (use this filename pattern for input configurations)
+- checkponts/*.cp (checkpoint files - information about monitored input files)
+
+
+See more about the configurations: [Log Search properties](logsearch_properties.md), [Log Feeder properties](logfeeder_properties.md) and [Log Feeder shipper configurations](shipper_configurations.md)
+
+#### Starting Log Search / Log Feeders
+
+Go to Ambari UI and start Log Search components or manually starts by `logsearch.sh` or `logfeeder.sh` script
+```bash
+# starting Log Search
+/usr/lib/ambari-logsearch-portal/bin/logsearch.sh start
+# starting Log Feeder
+/usr/lib/ambari-logsearch-logfeeder/bin/logsearch.sh start
+```
\ No newline at end of file
diff --git a/docs/schema_fields.md b/docs/schema_fields.md
deleted file mode 100644
index 4fb1258..0000000
--- a/docs/schema_fields.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!---
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index a4d46a5..6a2a7a0 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -15,15 +15,16 @@
site_name: Ambari Log Search
site_dir: mkdocs-site
nav:
-- Home: index.md
+- Overview: index.md
- User Guide:
- - Getting Started: getting_started.md
- - Add New Service: add_new_input.md
- - Schema Fields: schema_fields.md
+ - Installation: installation.md
+ - Collections: collections.md
+ - Add New Logs: add_new_input.md
- Cloud Mode: cloud_mode.md
+ - Development: development.md
- Configuration:
- Log Search properties: logsearch_properties.md
- Log Feeder properties: logfeeder_properties.md
- Shipper configurations: shipper_configurations.md
-- Docs: docs.md
+- API Docs: docs.md
- About: about.md
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index 6a3157b..c810257 100644
--- a/pom.xml
+++ b/pom.xml
@@ -23,6 +23,7 @@
<packaging>pom</packaging>
<name>Ambari Logsearch Parent</name>
<url>http://maven.apache.org</url>
+ <description>Ambari Logsearch Parent</description>
<profiles>
<profile>
<id>all</id>
@@ -361,6 +362,8 @@
<configuration>
<excludes>
<exclude>LICENSE</exclude>
+ <exclude>**/*.jpg</exclude>
+ <exclude>**/*.png</exclude>
<exclude>**/*.md</exclude>
<exclude>**/*.json</exclude>
<exclude>**/*.log</exclude>
