Move to using cloc 1.76 as minimum
- Change minimum required version of cloc for counting to 1.76
- Start utilizing multiprocessing for cloc
- Fix Readme to reflect this change, note that it's optional
diff --git a/README.md b/README.md
index dfdeccd..6797af7 100644
--- a/README.md
+++ b/README.md
@@ -65,19 +65,12 @@
## Requirements:
- - [cloc](https://github.com/AlDanial/cloc) version 1.70 or later
+ - [cloc](https://github.com/AlDanial/cloc) version 1.76 or later `(optional)`
- git binaries
- python3 (3.3 or later)
- python3-elasticsearch
- python3-dateutils
-
-### Multi-threaded CLoC counter
- To speed up things, you can use the multi-threaded version of cloc
- found at [this location](https://raw.githubusercontent.com/stsnel/cloc/multithread/cloc).
- On machines with many cores, you may experience a tenfold speed increase in
- the SLoC and evolution scans.
- It requires ForkManager, which may be installed via CPAN:
- `cpan Parallel::ForkManager`
+
# Get involved
TBD. Please see https://kibble.apache.org/ for details!
diff --git a/src/plugins/utils/sloc.py b/src/plugins/utils/sloc.py
index 25f1a5b..38c0e7d 100644
--- a/src/plugins/utils/sloc.py
+++ b/src/plugins/utils/sloc.py
@@ -22,10 +22,14 @@
import subprocess
import time
import re
+import multiprocessing
def count(path):
""" Count lines of Code """
- inp = subprocess.check_output("cloc --quiet --progress-rate=0 %s" % path, shell = True).decode('ascii', 'replace')
+ # We determine how many cores there are, and adjust the
+ # process count based on that. Max 4 procs.
+ my_core_count = min((4, int( multiprocessing.cpu_count() )))
+ inp = subprocess.check_output("cloc --quiet --progress-rate=0 --processes=%u %s" % (my_core_count, path), shell = True).decode('ascii', 'replace')
m = re.search(r".*Language\s+files\s+blank\s+comment\s+code[\s\S]+?-+([\s\S]+?)-+[\s\S]+?SUM:\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)", inp, flags=re.MULTILINE|re.UNICODE)
languages = {}
ccount = 0