IMPALA-9010: Add builtin mask functions
There're 6 builtin GenericUDFs for column masking in Hive:
mask_show_first_n(value, charCount, upperChar, lowerChar, digitChar,
otherChar, numberChar)
mask_show_last_n(value, charCount, upperChar, lowerChar, digitChar,
otherChar, numberChar)
mask_first_n(value, charCount, upperChar, lowerChar, digitChar,
otherChar, numberChar)
mask_last_n(value, charCount, upperChar, lowerChar, digitChar,
otherChar, numberChar)
mask_hash(value)
mask(value, upperChar, lowerChar, digitChar, otherChar, numberChar,
dayValue, monthValue, yearValue)
Description of the parameters:
value - value to mask. Supported types: TINYINT, SMALLINT, INT,
BIGINT, STRING, VARCHAR, CHAR, DATE(only for mask()).
charCount - number of characters. Default value: 4
upperChar - character to replace upper-case characters with. Specify
-1 to retain original character. Default value: 'X'
lowerChar - character to replace lower-case characters with. Specify
-1 to retain original character. Default value: 'x'
digitChar - character to replace digit characters with. Specify -1
to retain original character. Default value: 'n'
otherChar - character to replace all other characters with. Specify
-1 to retain original character. Default value: -1
numberChar - character to replace digits in a number with. Valid
values: 0-9. Default value: '1'
dayValue - value to replace day field in a date with.
Specify -1 to retain original value. Valid values: 1-31.
Default value: 1
monthValue - value to replace month field in a date with. Specify -1
to retain original value. Valid values: 0-11. Default
value: 0
yearValue - value to replace year field in a date with. Specify -1
to retain original value. Default value: 1
In Hive, these functions accept variable length of arguments in
non-restricted types:
mask_show_first_n(val)
mask_show_first_n(val, 8)
mask_show_first_n(val, 8, 'X', 'x', 'n')
mask_show_first_n(val, 8, 'x', 'x', 'x', 'x', 2)
mask_show_first_n(val, 8, 'x', -1, 'x', 'x', '9')
The arguments of upperChar, lowerChar, digitChar, otherChar and
numberChar can be in string or numeric types.
Impala doesn't support Hive GenericUDFs, so we are lack of these mask
functions to support Ranger column masking policies. On the other hand,
we want the masking functions to be evaluated in the C++ builtin logic
rather than calling out to java UDFs for performance. This patch
introduces our builtin implementation of them.
We currently don't have a corresponding framework for GenericUDF
(IMPALA-9271), so we implement these by overloads. However, it may
requires hundreds of overloads to cover all possible combinations. We
just implement some important overloads, including
- those used by Ranger default masking policies,
- those with simple arguments and may be useful for users,
- an overload with all arguments in int type for full functionality.
Char argument need to be converted to their ASCII value.
Tests:
- Add BE tests in expr-test
Change-Id: Ica779a1bf63a085d51f3b533f654cbaac102a664
Reviewed-on: http://gerrit.cloudera.org:8080/14963
Reviewed-by: Quanlong Huang <huangquanlong@gmail.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
diff --git a/be/src/codegen/impala-ir.cc b/be/src/codegen/impala-ir.cc
index 1459316..dce7840 100644
--- a/be/src/codegen/impala-ir.cc
+++ b/be/src/codegen/impala-ir.cc
@@ -49,6 +49,7 @@
#include "exprs/in-predicate-ir.cc"
#include "exprs/is-null-predicate-ir.cc"
#include "exprs/like-predicate-ir.cc"
+#include "exprs/mask-functions-ir.cc"
#include "exprs/math-functions-ir.cc"
#include "exprs/operators-ir.cc"
#include "exprs/scalar-expr-ir.cc"
diff --git a/be/src/exprs/CMakeLists.txt b/be/src/exprs/CMakeLists.txt
index 33e081b..b80d77c 100644
--- a/be/src/exprs/CMakeLists.txt
+++ b/be/src/exprs/CMakeLists.txt
@@ -48,6 +48,7 @@
like-predicate.cc
like-predicate-ir.cc
literal.cc
+ mask-functions-ir.cc
math-functions-ir.cc
null-literal.cc
operators-ir.cc
diff --git a/be/src/exprs/expr-test.cc b/be/src/exprs/expr-test.cc
index f76d620..69dfb50 100644
--- a/be/src/exprs/expr-test.cc
+++ b/be/src/exprs/expr-test.cc
@@ -10076,6 +10076,407 @@
TestErrorString("get_json_object('[1,2,3]', ' ')",
"Failed to parse json path ' ': Should start with '$'\n");
}
+
+TEST_P(ExprTest, MaskShowFirstNTest) {
+ // Test overrides for string value.
+ // Replace upper case with 'X', lower case with 'x', digit chars with '0', other chars
+ // with ':'.
+ TestStringValue("mask_show_first_n('TestString-123', 4, 'X', 'x', '0', ':')",
+ "TestXxxxxx:000");
+ TestStringValue(
+ "mask_show_first_n(cast('TestString-123' as varchar(24)), 4, 'X', 'x', '0', ':')",
+ "TestXxxxxx:000");
+ TestStringValue(
+ "mask_show_first_n(cast('TestString-123' as char(24)), 4, 'X', 'x', '0', ':')",
+ "TestXxxxxx:000::::::::::");
+ TestStringValue("mask_show_first_n('')", "");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#')", "abcdxXXXXXnnnnn-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 0)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', -1)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 2)", "abxxxXXXXXnnnnn-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 100)", "abcdeABCDE12345-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 3, '*', '*', '*', '*', 1)",
+ "abc**************"); // The last argument 1 is unused since the value is string.
+ // Most importantly, test the override used by Ranger transformer.
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 4, 'x', 'x', 'x', -1, '1')",
+ "abcdxxxxxxxxxxx-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 0, '*', '*', '*', -1, '9')",
+ "***************-#");
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 0, 65, 97, 48, -1, 9)",
+ "aaaaaAAAAA00000-#");
+ TestStringValue("mask_show_first_n('abcdeABCDE12345-#', 4, -1, -1, -1, -1, 9)",
+ "abcdeABCDE12345-#");
+
+ // Test overrides for numeric value.
+ TestValue("mask_show_first_n(0)", TYPE_BIGINT, 0);
+ TestValue("mask_show_first_n(0, 0)", TYPE_BIGINT, 1);
+ TestValue("mask_show_first_n(0, 0, -1, -1, -1, -1, 5)", TYPE_BIGINT, 5);
+ TestValue("mask_show_first_n(cast(123 as tinyint), 2, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 125);
+ TestValue("mask_show_first_n(cast(12345 as smallint), 3, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 12355);
+ TestValue("mask_show_first_n(cast(12345 as int), 0, 'x', 'x', 'x', 'x', 5)",
+ TYPE_BIGINT, 55555);
+ TestValue("mask_show_first_n(cast(12345 as bigint), 4, -1, -1, -1, -1, -1)",
+ TYPE_BIGINT, 12341);
+ TestValue("mask_show_first_n(123456789)", TYPE_BIGINT, 123411111);
+ TestValue("mask_show_first_n(123456789, 0)", TYPE_BIGINT, 111111111);
+ TestValue("mask_show_first_n(123456789, -1)", TYPE_BIGINT, 111111111);
+ TestValue("mask_show_first_n(123456789, 2)", TYPE_BIGINT, 121111111);
+ TestValue("mask_show_first_n(123456789, 100)", TYPE_BIGINT, 123456789);
+ TestValue("mask_show_first_n(123456789, 3, '*', '*', '*', '*', 2)", TYPE_BIGINT,
+ 123222222); // Only the last mask argument 2 is unused since the value is numeric.
+ // Most importantly, test the override used by Ranger transformer.
+ TestValue("mask_show_first_n(12345678900, 4, 'x', 'x', 'x', -1, '1')", TYPE_BIGINT,
+ 12341111111);
+ TestValue("mask_show_first_n(12345678900, 0, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ 99999999999);
+ TestValue("mask_show_first_n(-12345678900, 0, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ -99999999999);
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestValue("mask_show_first_n(12345678, 0, 65, 97, 48, -1, 9)", TYPE_BIGINT, 99999999);
+ // Illegal masked_number (not in [0, 9]) is converted to default value 1.
+ TestValue("mask_show_first_n(12345678, 4, -1, -1, -1, -1, 10)", TYPE_BIGINT, 12341111);
+ TestValue("mask_show_first_n(12345678, 4, -1, -1, -1, -1, -1)", TYPE_BIGINT, 12341111);
+
+ // Error handling
+ // Empty chars are converted to default values. Pick the first char for long strings.
+ TestStringValue("mask_show_first_n('TestString-123', 4, '', 'bBBBB', '', 'dDDD')",
+ "TestXbbbbbdnnn");
+ TestIsNull("mask_show_first_n(cast(NULL as STRING))", TYPE_STRING);
+ TestIsNull("mask_show_first_n(cast(NULL as BIGINT))", TYPE_BIGINT);
+ TestErrorString("mask_show_first_n(123456789, 4, 'aa', 'bb', 'cc', -1, 'a')",
+ "Can't convert 'a' to a valid masked_number. Valid values: 0-9.\n");
+ TestErrorString("mask_show_first_n(123456789, 4, 'aa', 'bb', 'cc', -1, '10')",
+ "Can't convert '10' to a valid masked_number. Valid values: 0-9.\n");
+}
+
+TEST_P(ExprTest, MaskShowLastNTest) {
+ // Test overrides for string value.
+ // Replace upper case with 'X', lower case with 'x', digit chars with '0', other chars
+ // with ':'.
+ TestStringValue("mask_show_last_n('TestString-123', 4, 'X', 'x', '0', ':')",
+ "XxxxXxxxxx-123");
+ TestStringValue(
+ "mask_show_last_n(cast('TestString-123' as varchar(24)), 4, 'X', 'x', '0', ':')",
+ "XxxxXxxxxx-123");
+ TestStringValue(
+ "mask_show_last_n(cast('TestString-123' as char(24)), 4, 'X', 'x', '0', ':')",
+ "XxxxXxxxxx:000:::::: "); // Remain the last 4 whitespaces
+ TestStringValue("mask_show_last_n('')", "");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#')", "xxxxxXXXXXnnn45-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 0)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', -1)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 2)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 100)", "abcdeABCDE12345-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 3, '*', '*', '*', '*', 1)",
+ "**************5-#"); // The last argument 1 is unused since the value is string.
+ // Most importantly, test the override used by Ranger transformer.
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 4, 'x', 'x', 'x', -1, '1')",
+ "xxxxxxxxxxxxx45-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 0, '*', '*', '*', -1, '9')",
+ "***************-#");
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 0, 65, 97, 48, -1, 9)",
+ "aaaaaAAAAA00000-#");
+ TestStringValue("mask_show_last_n('abcdeABCDE12345-#', 4, -1, -1, -1, -1, 9)",
+ "abcdeABCDE12345-#");
+
+ // Test overrides for numeric value.
+ TestValue("mask_show_last_n(0)", TYPE_BIGINT, 0);
+ TestValue("mask_show_last_n(0, 0)", TYPE_BIGINT, 1);
+ TestValue("mask_show_last_n(0, 0, -1, -1, -1, -1, 5)", TYPE_BIGINT, 5);
+ TestValue("mask_show_last_n(cast(123 as tinyint), 2, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 523);
+ TestValue("mask_show_last_n(cast(12345 as smallint), 3, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 55345);
+ TestValue("mask_show_last_n(cast(12345 as int), 0, 'x', 'x', 'x', 'x', 5)",
+ TYPE_BIGINT, 55555);
+ TestValue("mask_show_last_n(cast(12345 as bigint), 4, -1, -1, -1, -1, -1)",
+ TYPE_BIGINT, 12345);
+ TestValue("mask_show_last_n(123456789)", TYPE_BIGINT, 111116789);
+ TestValue("mask_show_last_n(123456789, 0)", TYPE_BIGINT, 111111111);
+ TestValue("mask_show_last_n(123456789, -1)", TYPE_BIGINT, 111111111);
+ TestValue("mask_show_last_n(123456789, 2)", TYPE_BIGINT, 111111189);
+ TestValue("mask_show_last_n(123456789, 100)", TYPE_BIGINT, 123456789);
+ TestValue("mask_show_last_n(123456789, 3, '*', '*', '*', '*', 2)", TYPE_BIGINT,
+ 222222789); // Only the last mask argument 2 is unused since the value is numeric.
+ // Most importantly, test the override used by Ranger transformer.
+ TestValue("mask_show_last_n(12345678900, 4, 'x', 'x', 'x', -1, '1')", TYPE_BIGINT,
+ 11111118900);
+ TestValue("mask_show_last_n(12345678900, 0, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ 99999999999);
+ TestValue("mask_show_last_n(-12345678900, 0, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ -99999999999);
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestValue("mask_show_last_n(12345678, 0, 65, 97, 48, -1, 9)", TYPE_BIGINT, 99999999);
+ // Illegal masked_number (not in [0, 9]) is converted to default value 1.
+ TestValue("mask_show_last_n(12345678, 4, -1, -1, -1, -1, 10)", TYPE_BIGINT, 11115678);
+ TestValue("mask_show_last_n(12345678, 4, -1, -1, -1, -1, -1)", TYPE_BIGINT, 11115678);
+
+ // Error handling
+ // Empty chars are converted to default values. Pick the first char for long strings.
+ TestStringValue("mask_show_last_n('TestString-123', 4, '', 'bBBBB', '', 'dDDD')",
+ "XbbbXbbbbb-123");
+ TestIsNull("mask_show_last_n(cast(NULL as STRING))", TYPE_STRING);
+ TestIsNull("mask_show_last_n(cast(NULL as BIGINT))", TYPE_BIGINT);
+ TestErrorString("mask_show_last_n(123456789, 4, 'aa', 'bb', 'cc', -1, 'a')",
+ "Can't convert 'a' to a valid masked_number. Valid values: 0-9.\n");
+ TestErrorString("mask_show_last_n(123456789, 4, 'aa', 'bb', 'cc', -1, '10')",
+ "Can't convert '10' to a valid masked_number. Valid values: 0-9.\n");
+}
+
+TEST_P(ExprTest, MaskFirstNTest) {
+ // Test overrides for string value.
+ // Replace upper case with 'X', lower case with 'x', digit chars with '0', other chars
+ // with ':'.
+ TestStringValue("mask_first_n('TestString-123', 4, 'X', 'x', '0', ':')",
+ "XxxxString-123");
+ TestStringValue(
+ "mask_first_n(cast('TestString-123' as varchar(24)), 4, 'X', 'x', '0', ':')",
+ "XxxxString-123");
+ TestStringValue(
+ "mask_first_n(cast('TestString-123' as char(24)), 4, 'X', 'x', '0', ':')",
+ "XxxxString-123 ");
+ TestStringValue("mask_first_n('')", "");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#')", "xxxxeABCDE12345-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 0)", "abcdeABCDE12345-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', -1)", "abcdeABCDE12345-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 2)", "xxcdeABCDE12345-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 100)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 3, '*', '*', '*', '*', 1)",
+ "***deABCDE12345-#"); // The last argument 1 is unused since the value is string.
+ // Most importantly, test the override used by Ranger transformer.
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 4, 'x', 'x', 'x', -1, '1')",
+ "xxxxeABCDE12345-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 0, '*', '*', '*', -1, '9')",
+ "abcdeABCDE12345-#");
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 100, 65, 97, 48, -1, 9)",
+ "aaaaaAAAAA00000-#");
+ TestStringValue("mask_first_n('abcdeABCDE12345-#', 100, -1, -1, -1, -1, 9)",
+ "abcdeABCDE12345-#");
+
+ // Test overrides for numeric value.
+ TestValue("mask_first_n(0)", TYPE_BIGINT, 1);
+ TestValue("mask_first_n(0, 0)", TYPE_BIGINT, 0);
+ TestValue("mask_first_n(0, 4, -1, -1, -1, -1, 5)", TYPE_BIGINT, 5);
+ TestValue("mask_first_n(cast(123 as tinyint), 2, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 553);
+ TestValue("mask_first_n(cast(12345 as smallint), 3, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 55545);
+ TestValue("mask_first_n(cast(12345 as int), 0, 'x', 'x', 'x', 'x', 5)",
+ TYPE_BIGINT, 12345);
+ TestValue("mask_first_n(cast(12345 as bigint), 4, -1, -1, -1, -1, -1)",
+ TYPE_BIGINT, 11115);
+ TestValue("mask_first_n(123456789)", TYPE_BIGINT, 111156789);
+ TestValue("mask_first_n(123456789, 0)", TYPE_BIGINT, 123456789);
+ TestValue("mask_first_n(123456789, -1)", TYPE_BIGINT, 123456789);
+ TestValue("mask_first_n(123456789, 2)", TYPE_BIGINT, 113456789);
+ TestValue("mask_first_n(123456789, 100)", TYPE_BIGINT, 111111111);
+ TestValue("mask_first_n(123456789, 3, '*', '*', '*', '*', 2)", TYPE_BIGINT,
+ 222456789); // Only the last mask argument 2 is unused since the value is numeric.
+ // Most importantly, test the override used by Ranger transformer.
+ TestValue("mask_first_n(12345678900, 4, 'x', 'x', 'x', -1, '1')", TYPE_BIGINT,
+ 11115678900);
+ TestValue("mask_first_n(12345678900, 20, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ 99999999999);
+ TestValue("mask_first_n(-12345678900, 20, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ -99999999999);
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestValue("mask_first_n(12345678, 10, 65, 97, 48, -1, 9)", TYPE_BIGINT, 99999999);
+ // Illegal masked_number (not in [0, 9]) is converted to default value 1.
+ TestValue("mask_first_n(12345678, 4, -1, -1, -1, -1, 10)", TYPE_BIGINT, 11115678);
+ TestValue("mask_first_n(12345678, 4, -1, -1, -1, -1, -1)", TYPE_BIGINT, 11115678);
+
+ // Error handling
+ // Empty chars are converted to default values. Pick the first char for long strings.
+ TestStringValue("mask_first_n('TestString-123', 4, '', 'bBBBB', '', 'dDDD')",
+ "XbbbString-123");
+ TestIsNull("mask_first_n(cast(NULL as STRING))", TYPE_STRING);
+ TestIsNull("mask_first_n(cast(NULL as BIGINT))", TYPE_BIGINT);
+ TestErrorString("mask_first_n(123456789, 4, 'aa', 'bb', 'cc', -1, 'a')",
+ "Can't convert 'a' to a valid masked_number. Valid values: 0-9.\n");
+ TestErrorString("mask_first_n(123456789, 4, 'aa', 'bb', 'cc', -1, '10')",
+ "Can't convert '10' to a valid masked_number. Valid values: 0-9.\n");
+}
+
+TEST_P(ExprTest, MaskLastNTest) {
+ // Test overrides for string value.
+ // Replace upper case with 'X', lower case with 'x', digit chars with '0', other chars
+ // with ':'.
+ TestStringValue("mask_last_n('TestString-123', 4, 'X', 'x', '0', ':')",
+ "TestString:000");
+ TestStringValue(
+ "mask_last_n(cast('TestString-123' as varchar(24)), 4, 'X', 'x', '0', ':')",
+ "TestString:000");
+ TestStringValue(
+ "mask_last_n(cast('TestString-123' as char(24)), 4, 'X', 'x', '0', ':')",
+ "TestString-123 ::::");
+ TestStringValue("mask_last_n('')", "");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#')", "abcdeABCDE123nn-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 0)", "abcdeABCDE12345-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', -1)", "abcdeABCDE12345-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 2)", "abcdeABCDE12345-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 100)", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 3, '*', '*', '*', '*', 1)",
+ "abcdeABCDE1234***"); // The last argument 1 is unused since the value is string.
+ // Most importantly, test the override used by Ranger transformer.
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 4, 'x', 'x', 'x', -1, '1')",
+ "abcdeABCDE123xx-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 100, '*', '*', '*', -1, '9')",
+ "***************-#");
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 100, 65, 97, 48, -1, 9)",
+ "aaaaaAAAAA00000-#");
+ TestStringValue("mask_last_n('abcdeABCDE12345-#', 100, -1, -1, -1, -1, 9)",
+ "abcdeABCDE12345-#");
+
+ // Test overrides for numeric value.
+ TestValue("mask_last_n(0)", TYPE_BIGINT, 1);
+ TestValue("mask_last_n(0, 0)", TYPE_BIGINT, 0);
+ TestValue("mask_last_n(0, 4, -1, -1, -1, -1, 5)", TYPE_BIGINT, 5);
+ TestValue("mask_last_n(cast(123 as tinyint), 2, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 155);
+ TestValue("mask_last_n(cast(12345 as smallint), 3, 'x', 'x', 'x', -1, '5')",
+ TYPE_BIGINT, 12555);
+ TestValue("mask_last_n(cast(12345 as int), 10, 'x', 'x', 'x', 'x', 5)",
+ TYPE_BIGINT, 55555);
+ TestValue("mask_last_n(cast(12345 as bigint), 4, -1, -1, -1, -1, -1)",
+ TYPE_BIGINT, 11111);
+ TestValue("mask_last_n(123456789)", TYPE_BIGINT, 123451111);
+ TestValue("mask_last_n(123456789, 0)", TYPE_BIGINT, 123456789);
+ TestValue("mask_last_n(123456789, -1)", TYPE_BIGINT, 123456789);
+ TestValue("mask_last_n(123456789, 2)", TYPE_BIGINT, 123456711);
+ TestValue("mask_last_n(123456789, 100)", TYPE_BIGINT, 111111111);
+ TestValue("mask_last_n(123456789, 3, '*', '*', '*', '*', 2)", TYPE_BIGINT,
+ 123456222); // Only the last mask argument 2 is unused since the value is numeric.
+ // Most importantly, test the override used by Ranger transformer.
+ TestValue("mask_last_n(12345678900, 4, 'x', 'x', 'x', -1, '1')", TYPE_BIGINT,
+ 12345671111);
+ TestValue("mask_last_n(12345678900, 20, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ 99999999999);
+ TestValue("mask_last_n(-12345678900, 20, '*', '*', '*', -1, '9')", TYPE_BIGINT,
+ -99999999999);
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestValue("mask_last_n(12345678, 10, 65, 97, 48, -1, 9)", TYPE_BIGINT, 99999999);
+ // Illegal masked_number (not in [0, 9]) is converted to default value 1.
+ TestValue("mask_last_n(12345678, 4, -1, -1, -1, -1, 10)", TYPE_BIGINT, 12341111);
+ TestValue("mask_last_n(12345678, 4, -1, -1, -1, -1, -1)", TYPE_BIGINT, 12341111);
+
+ // Error handling
+ // Empty chars are converted to default values. Pick the first char for long strings.
+ TestStringValue("mask_last_n('TestString-123', 4, '', 'bBBBB', '', 'dDDD')",
+ "TestStringdnnn");
+ TestIsNull("mask_last_n(cast(NULL as STRING))", TYPE_STRING);
+ TestIsNull("mask_last_n(cast(NULL as BIGINT))", TYPE_BIGINT);
+ TestErrorString("mask_last_n(123456789, 4, 'aa', 'bb', 'cc', -1, 'a')",
+ "Can't convert 'a' to a valid masked_number. Valid values: 0-9.\n");
+ TestErrorString("mask_last_n(123456789, 4, 'aa', 'bb', 'cc', -1, '10')",
+ "Can't convert '10' to a valid masked_number. Valid values: 0-9.\n");
+}
+
+TEST_P(ExprTest, MaskTest) {
+ // Test overrides for string value.
+ // Replace upper case with 'X', lower case with 'x', digit chars with '0', other chars
+ // with ':'.
+ TestStringValue("mask('TestString-123', 'X', 'x', '0', ':')", "XxxxXxxxxx:000");
+ TestStringValue("mask(cast('TestString-123' as varchar(24)), 'X', 'x', '0', ':')",
+ "XxxxXxxxxx:000");
+ TestStringValue("mask(cast('TestString-123' as char(24)), 'X', 'x', '0', ':')",
+ "XxxxXxxxxx:000::::::::::");
+ TestStringValue("mask('')", "");
+ TestStringValue("mask('abcdeABCDE12345-#')", "xxxxxXXXXXnnnnn-#");
+ TestStringValue("mask('abcdeABCDE12345-#', '*', '*', '*', '*', 1)",
+ "*****************"); // The last argument 1 is unused since the value is string.
+ // Most importantly, test the override used by Ranger transformer.
+ TestStringValue("mask('abcdeABCDE12345-#', 'x', 'x', 'x', -1, '1')",
+ "xxxxxxxxxxxxxxx-#");
+ TestStringValue("mask('abcdeABCDE12345-#', '*', '*', '*', -1, '9')",
+ "***************-#");
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestStringValue("mask('abcdeABCDE12345-#', 65, 97, 48, -1, 9)", "aaaaaAAAAA00000-#");
+ TestStringValue("mask('abcdeABCDE12345-#', -1, -1, -1, -1, 9)", "abcdeABCDE12345-#");
+
+ // Test overrides for numeric value.
+ TestValue("mask(0)", TYPE_BIGINT, 1);
+ TestValue("mask(0, -1, -1, -1, -1, 5)", TYPE_BIGINT, 5);
+ TestValue("mask(cast(123 as tinyint), 'x', 'x', 'x', -1, '5')", TYPE_BIGINT, 555);
+ TestValue("mask(cast(12345 as smallint), 'x', 'x', 'x', -1, '5')", TYPE_BIGINT, 55555);
+ TestValue("mask(cast(12345 as int), 'x', 'x', 'x', 'x', 5)", TYPE_BIGINT, 55555);
+ TestValue("mask(cast(12345 as bigint), -1, -1, -1, -1, -1)", TYPE_BIGINT, 11111);
+ TestValue("mask(12345678900)", TYPE_BIGINT, 11111111111);
+ TestValue("mask(12345678900, '*', '*', '*', '*', 2)", TYPE_BIGINT, 22222222222);
+ // Most importantly, test the override used by Ranger transformer.
+ TestValue("mask(12345678900, 'x', 'x', 'x', -1, '1')", TYPE_BIGINT, 11111111111);
+ TestValue("mask(12345678900, '*', '*', '*', -1, '9')", TYPE_BIGINT, 99999999999);
+ TestValue("mask(-12345678900, '*', '*', '*', -1, '9')", TYPE_BIGINT, -99999999999);
+ // Test all int arguments. 65 = 'A', 97 = 'a', 48 = '0'.
+ TestValue("mask(12345678, 65, 97, 48, -1, 9)", TYPE_BIGINT, 99999999);
+ // Illegal masked_number (not in [0, 9]) is converted to default value 1.
+ TestValue("mask(12345678, -1, -1, -1, -1, 10)", TYPE_BIGINT, 11111111);
+ TestValue("mask(12345678, -1, -1, -1, -1, -1)", TYPE_BIGINT, 11111111);
+
+ // Test overrides for Date value.
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 11, 2, 2020)",
+ DateValue(2020, 3, 11));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 32, 12, 2020)",
+ DateValue(2020, 1, 1));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, -1, -1, -1)",
+ DateValue(2019, 12, 31));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 10, -1, -1)",
+ DateValue(2019, 12, 10));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, -1, 0, -1)",
+ DateValue(2019, 1, 31));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, -1, -1, 10)",
+ DateValue(10, 12, 31));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 10, 0, -1)",
+ DateValue(2019, 1, 10));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 10, -1, 10)",
+ DateValue(10, 12, 10));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, -1, 0, 10)",
+ DateValue(10, 1, 31));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 10, 0, 10)",
+ DateValue(10, 1, 10));
+
+ // Error handling
+ // Empty chars are converted to default values. Pick the first char for long strings.
+ TestStringValue("mask('TestString-123', '', 'bBBBB', '', 'dDDD')", "XbbbXbbbbbdnnn");
+ TestIsNull("mask(cast(NULL as STRING))", TYPE_STRING);
+ TestIsNull("mask(cast(NULL as BIGINT))", TYPE_BIGINT);
+ TestIsNull("mask(cast(NULL as DATE))", TYPE_DATE);
+ TestErrorString("mask(123456789, 'aa', 'bb', 'cc', -1, 'a')",
+ "Can't convert 'a' to a valid masked_number. Valid values: 0-9.\n");
+ TestErrorString("mask(123456789, 'aa', 'bb', 'cc', -1, '10')",
+ "Can't convert '10' to a valid masked_number. Valid values: 0-9.\n");
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 10, 0, 0)",
+ DateValue(1, 1, 10));
+ TestDateValue("mask(cast('2019-12-31' as date), -1, -1, -1, -1, -1, 10, 0, 10000)",
+ DateValue(1, 1, 10));
+ // This is a different behavior than Hive's. Hive will convert '2019-02-30' to
+ // '2019-03-02', while Impala converts it to NULL.
+ TestIsNull("mask(cast('2019-02-03' as date), -1, -1, -1, -1, -1, 30, -1, -1)",
+ TYPE_DATE);
+}
+
+TEST_P(ExprTest, MaskHashTest) {
+ TestStringValue("mask_hash('TestString-123')",
+ "8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90");
+ TestStringValue("mask_hash(cast('TestString-123' as varchar(24)))",
+ "8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90");
+ TestStringValue("mask_hash(cast('TestString-123' as char(24)))",
+ "30a88603135d3a6f7a66b4f9193da1ab4423aed45fb8fe736c2f2a08977f2bdd");
+ TestIsNull("mask_hash(cast(123 as tinyint))", TYPE_BIGINT);
+ TestIsNull("mask_hash(cast(12345 as smallint))", TYPE_BIGINT);
+ TestIsNull("mask_hash(cast(12345 as int))", TYPE_BIGINT);
+ TestIsNull("mask_hash(cast(12345 as bigint))", TYPE_BIGINT);
+ TestIsNull("mask_hash(cast(1 as boolean))", TYPE_BOOLEAN);
+ TestIsNull("mask_hash(cast(12345 as double))", TYPE_DOUBLE);
+ TestIsNull("mask_hash(cast('2016-04-20' as date))", TYPE_DATE);
+ TestIsNull("mask_hash(cast('2016-04-20' as timestamp))", TYPE_TIMESTAMP);
+}
+
} // namespace impala
INSTANTIATE_TEST_CASE_P(Instantiations, ExprTest, ::testing::Values(
diff --git a/be/src/exprs/mask-functions-ir.cc b/be/src/exprs/mask-functions-ir.cc
new file mode 100644
index 0000000..15177bf
--- /dev/null
+++ b/be/src/exprs/mask-functions-ir.cc
@@ -0,0 +1,726 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/mask-functions.h"
+
+#include <gutil/strings/substitute.h>
+#include <openssl/err.h>
+#include <openssl/sha.h>
+
+#include "exprs/anyval-util.h"
+#include "exprs/math-functions.h"
+#include "exprs/string-functions.h"
+#include "util/ubsan.h"
+
+#include "common/names.h"
+
+using namespace impala;
+using namespace impala_udf;
+
+const static int CHAR_COUNT = 4;
+const static int MASKED_UPPERCASE = 'X';
+const static int MASKED_LOWERCASE = 'x';
+const static int MASKED_DIGIT = 'n';
+const static int MASKED_OTHER_CHAR = -1;
+const static int MASKED_NUMBER = 1;
+const static int MASKED_DAY_COMPONENT_VAL = 1;
+const static int MASKED_MONTH_COMPONENT_VAL = 0;
+const static int MASKED_YEAR_COMPONENT_VAL = 1;
+const static int UNMASKED_VAL = -1;
+
+/// Mask the given char depending on its type. UNMASKED_VAL(-1) means keeping the
+/// original value.
+static inline uint8_t MaskTransform(uint8_t val, int masked_upper_char,
+ int masked_lower_char, int masked_digit_char, int masked_other_char) {
+ if ('A' <= val && val <= 'Z') {
+ if (masked_upper_char == UNMASKED_VAL) return val;
+ return masked_upper_char;
+ }
+ if ('a' <= val && val <= 'z') {
+ if (masked_lower_char == UNMASKED_VAL) return val;
+ return masked_lower_char;
+ }
+ if ('0' <= val && val <= '9') {
+ if (masked_digit_char == UNMASKED_VAL) return val;
+ return masked_digit_char;
+ }
+ if (masked_other_char == UNMASKED_VAL) return val;
+ return masked_other_char;
+}
+
+/// Mask the substring in range [start, end) of the given string value. Using rules in
+/// 'MaskTransform'.
+static StringVal MaskSubStr(FunctionContext* ctx, const StringVal& val,
+ int start, int end, int masked_upper_char, int masked_lower_char,
+ int masked_digit_char, int masked_other_char) {
+ DCHECK_GE(start, 0);
+ DCHECK_LT(start, end);
+ DCHECK_LE(end, val.len);
+ StringVal result(ctx, val.len);
+ if (UNLIKELY(result.is_null)) return StringVal::null();
+ Ubsan::MemCpy(result.ptr, val.ptr, start);
+ if (end < val.len) Ubsan::MemCpy(result.ptr + end, val.ptr + end, val.len - end);
+ for (int i = start; i < end; ++i) {
+ result.ptr[i] = MaskTransform(val.ptr[i], masked_upper_char, masked_lower_char,
+ masked_digit_char, masked_other_char);
+ }
+ return result;
+}
+
+/// Mask the given string except the first 'un_mask_char_count' chars. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskShowFirstN.
+static inline StringVal MaskShowFirstNImpl(FunctionContext* ctx, const StringVal& val,
+ int un_mask_char_count, int masked_upper_char, int masked_lower_char,
+ int masked_digit_char, int masked_other_char) {
+ // To be consistent with Hive, negative char_count is treated as 0.
+ if (un_mask_char_count < 0) un_mask_char_count = 0;
+ if (val.is_null || val.len == 0 || un_mask_char_count >= val.len) return val;
+ return MaskSubStr(ctx, val, un_mask_char_count, val.len, masked_upper_char,
+ masked_lower_char, masked_digit_char, masked_other_char);
+}
+
+/// Mask the given string except the last 'un_mask_char_count' chars. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskShowLastN.
+static inline StringVal MaskShowLastNImpl(FunctionContext* ctx, const StringVal& val,
+ int un_mask_char_count, int masked_upper_char, int masked_lower_char,
+ int masked_digit_char, int masked_other_char) {
+ // To be consistent with Hive, negative char_count is treated as 0.
+ if (un_mask_char_count < 0) un_mask_char_count = 0;
+ if (val.is_null || val.len == 0 || un_mask_char_count >= val.len) return val;
+ return MaskSubStr(ctx, val, 0, val.len - un_mask_char_count, masked_upper_char,
+ masked_lower_char, masked_digit_char, masked_other_char);
+}
+
+/// Mask the first 'mask_char_count' chars of the given string. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskFirstN.
+static inline StringVal MaskFirstNImpl(FunctionContext* ctx, const StringVal& val,
+ int mask_char_count, int masked_upper_char, int masked_lower_char,
+ int masked_digit_char, int masked_other_char) {
+ if (mask_char_count <= 0 || val.is_null || val.len == 0) return val;
+ if (mask_char_count > val.len) mask_char_count = val.len;
+ return MaskSubStr(ctx, val, 0, mask_char_count, masked_upper_char,
+ masked_lower_char, masked_digit_char, masked_other_char);
+}
+
+/// Mask the last 'mask_char_count' chars of the given string. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskLastN.
+static inline StringVal MaskLastNImpl(FunctionContext* ctx, const StringVal& val,
+ int mask_char_count, int masked_upper_char, int masked_lower_char,
+ int masked_digit_char, int masked_other_char) {
+ if (mask_char_count <= 0 || val.is_null || val.len == 0) return val;
+ if (mask_char_count > val.len) mask_char_count = val.len;
+ return MaskSubStr(ctx, val, val.len - mask_char_count, val.len, masked_upper_char,
+ masked_lower_char, masked_digit_char, masked_other_char);
+}
+
+/// Mask the whole given string. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMask.
+static inline StringVal MaskImpl(FunctionContext* ctx, const StringVal& val,
+ int masked_upper_char, int masked_lower_char, int masked_digit_char,
+ int masked_other_char) {
+ if (val.is_null || val.len == 0) return val;
+ return MaskSubStr(ctx, val, 0, val.len, masked_upper_char,
+ masked_lower_char, masked_digit_char, masked_other_char);
+}
+
+static inline int GetNumDigits(int64_t val) {
+ if (val == 0) return 1;
+ if (val < 0) val = -val;
+ int num_digits = 0;
+ while (val != 0) {
+ num_digits++;
+ val /= 10;
+ }
+ return num_digits;
+}
+
+/// Mask the numeric value by replacing the digits with 'masked_number'. The first
+/// 'un_mask_char_count' digits will be kept unchanged. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskShowFirstN.
+static inline BigIntVal MaskShowFirstNImpl(const BigIntVal& val, int un_mask_digit_count,
+ int masked_number) {
+ if (val.is_null) return val;
+ // To be consistent with Hive, illegal masked_number is treated as default value.
+ if (masked_number < 0 || masked_number > 9) masked_number = MASKED_NUMBER;
+ if (un_mask_digit_count < 0) un_mask_digit_count = 0;
+ if (val.val == 0) return un_mask_digit_count > 0 ? 0 : masked_number;
+ int64_t unsigned_val = val.val;
+ // The sign won't be masked.
+ if (val.val < 0) unsigned_val = -unsigned_val;
+ int num_digits = GetNumDigits(unsigned_val);
+ // Number of digits to mask from the end
+ int mask_count = num_digits - un_mask_digit_count;
+ if (mask_count <= 0) return val;
+ int64_t result = 0;
+ int64_t base = 1;
+ for (int i = 0; i < mask_count; ++i) { // loop from end to start
+ result += masked_number * base;
+ base *= 10;
+ unsigned_val /= 10;
+ }
+ result += unsigned_val * base;
+ return {val.val < 0 ? -result : result};
+}
+
+/// Mask the numeric value by replacing the digits with 'masked_number'. The last
+/// 'un_mask_char_count' digits will be kept unchanged. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskShowLastN.
+static inline BigIntVal MaskShowLastNImpl(const BigIntVal& val, int un_mask_char_count,
+ int masked_number) {
+ if (val.is_null) return val;
+ // To be consistent with Hive, illegal masked_number is treated as default value.
+ if (masked_number < 0 || masked_number > 9) masked_number = MASKED_NUMBER;
+ if (un_mask_char_count < 0) un_mask_char_count = 0;
+ if (val.val == 0) return un_mask_char_count > 0 ? 0 : masked_number;
+ int64_t unsigned_val = val.val;
+ // The sign won't be masked.
+ if (val.val < 0) unsigned_val = -unsigned_val;
+ int num_digits = GetNumDigits(unsigned_val);
+ if (num_digits <= un_mask_char_count) return val;
+ int64_t base = 1;
+ for (int i = 0; i < un_mask_char_count; ++i) base *= 10;
+ int64_t result = unsigned_val % base;
+ for (int i = un_mask_char_count; i < num_digits; ++i) {
+ // It's possible that result overflows, e.g. val is 2^63-1 and masked_number is 9.
+ // Just continue with the overflowed result to be consistent with Hive.
+ result += masked_number * base;
+ base *= 10;
+ }
+ return {val.val < 0 ? -result : result};
+}
+
+/// Mask the numeric value by replacing the first 'mask_char_count' digits with
+/// 'masked_number'. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskFirstN.
+static inline BigIntVal MaskFirstNImpl(const BigIntVal& val, int mask_char_count,
+ int masked_number) {
+ if (val.is_null) return val;
+ int num_digits = GetNumDigits(val.val);
+ if (num_digits < mask_char_count) mask_char_count = num_digits;
+ return MaskShowLastNImpl(val, num_digits - mask_char_count, masked_number);
+}
+
+/// Mask the numeric value by replacing the last 'mask_char_count' digits with
+/// 'masked_number'. Ported from
+/// org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskLastN.
+static inline BigIntVal MaskLastNImpl(const BigIntVal& val, int mask_char_count,
+ int masked_number) {
+ if (val.is_null) return val;
+ int num_digits = GetNumDigits(val.val);
+ if (num_digits < mask_char_count) mask_char_count = num_digits;
+ return MaskShowFirstNImpl(val, num_digits - mask_char_count, masked_number);
+}
+
+/// Mask the Date value by replacing the day by 'day_value', the month by 'month_value',
+/// and the year by 'year_value'. UNMASKED_VAL(-1) means keeping the original value.
+/// Ported from org.apache.hadoop.hive.ql.udf.generic.GenericUDFMask.
+static DateVal MaskImpl(FunctionContext* ctx, const DateVal& val, int day_value,
+ int month_value, int year_value) {
+ if (val.is_null) return val;
+ if (day_value != UNMASKED_VAL && !(1 <= day_value && day_value <= 31)) {
+ day_value = MASKED_DAY_COMPONENT_VAL;
+ }
+ if (month_value != UNMASKED_VAL && !(0 <= month_value && month_value < 12)) {
+ month_value = MASKED_MONTH_COMPONENT_VAL;
+ }
+ if (year_value != UNMASKED_VAL && (year_value <= 0 || year_value >= 9999)) {
+ // Repalce illegal year for DateValue with default value.
+ year_value = MASKED_YEAR_COMPONENT_VAL;
+ }
+ int year, month, day;
+ DateValue dv = DateValue::FromDateVal(val);
+ // Extract year, month, day.
+ if (!dv.ToYearMonthDay(&year, &month, &day)) return DateVal::null();
+
+ if (year_value != UNMASKED_VAL) year = year_value;
+ // In DateValue, month starts from 1, so increase 'month_value' by 1.
+ if (month_value != UNMASKED_VAL) month = month_value + 1;
+ if (day_value != UNMASKED_VAL) day = day_value;
+ return DateValue(year, month, day).ToDateVal();
+}
+
+static inline uint8_t GetFirstChar(const StringVal& str, uint8_t default_value) {
+ // To be consistent with Hive, empty string is converted to default value. String with
+ // length > 1 will only use its first char.
+ return str.len == 0 ? default_value : str.ptr[0];
+}
+
+/// Get digit (masked_number) from StringVal. Only accept digits or -1.
+static inline bool GetDigitFromString(FunctionContext* ctx, const StringVal& str,
+ int* res) {
+ if (str.len != 1 || str.ptr[0] < '0' || str.ptr[0] > '9') {
+ ctx->SetError(Substitute(
+ "Can't convert '$0' to a valid masked_number. Valid values: 0-9.",
+ AnyValUtil::ToString(str)).c_str());
+ return false;
+ }
+ *res = str.ptr[0] - '0';
+ return true;
+}
+
+/// MaskShowFirstN overloads for string value
+StringVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const StringVal& val) {
+ return MaskShowFirstNImpl(ctx, val, CHAR_COUNT, MASKED_UPPERCASE, MASKED_LOWERCASE,
+ MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count) {
+ int un_mask_char_count = char_count.val;
+ return MaskShowFirstNImpl(ctx, val, un_mask_char_count, MASKED_UPPERCASE,
+ MASKED_LOWERCASE, MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char) {
+ return MaskShowFirstNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ GetFirstChar(other_char, MASKED_OTHER_CHAR));
+}
+StringVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return MaskShowFirstN(ctx, val, char_count, upper_char, lower_char, digit_char,
+ other_char);
+}
+StringVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ return MaskShowFirstNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ other_char.val);
+}
+StringVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return MaskShowFirstNImpl(ctx, val, char_count.val, upper_char.val, lower_char.val,
+ digit_char.val, other_char.val);
+}
+
+/// MaskShowFirstN overloads for numeric value
+BigIntVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val) {
+ return {MaskShowFirstNImpl(val, CHAR_COUNT, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count) {
+ return {MaskShowFirstNImpl(val, char_count.val, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return {MaskShowFirstNImpl(val, char_count.val, number_char.val)};
+}
+BigIntVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ int masked_number;
+ if (!GetDigitFromString(ctx, number_char, &masked_number)) return BigIntVal::null();
+ return {MaskShowFirstNImpl(val, char_count.val, masked_number)};
+}
+BigIntVal MaskFunctions::MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return {MaskShowFirstNImpl(val, char_count.val, number_char.val)};
+}
+
+/// MaskShowLastN overloads for string value
+StringVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const StringVal& val) {
+ return MaskShowLastNImpl(ctx, val, CHAR_COUNT, MASKED_UPPERCASE, MASKED_LOWERCASE,
+ MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count) {
+ int un_mask_char_count = char_count.val;
+ return MaskShowLastNImpl(ctx, val, un_mask_char_count, MASKED_UPPERCASE,
+ MASKED_LOWERCASE, MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char) {
+ return MaskShowLastNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ GetFirstChar(other_char, MASKED_OTHER_CHAR));
+}
+StringVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return MaskShowLastN(ctx, val, char_count, upper_char, lower_char, digit_char,
+ other_char);
+}
+StringVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ return MaskShowLastNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ other_char.val);
+}
+StringVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return MaskShowLastNImpl(ctx, val, char_count.val, upper_char.val, lower_char.val,
+ digit_char.val, other_char.val);
+}
+
+/// MaskShowLastN overloads for numeric value
+BigIntVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const BigIntVal& val) {
+ return {MaskShowLastNImpl(val, CHAR_COUNT, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count) {
+ return {MaskShowLastNImpl(val, char_count.val, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return {MaskShowLastNImpl(val, char_count.val, number_char.val)};
+}
+BigIntVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ int masked_number;
+ if (!GetDigitFromString(ctx, number_char, &masked_number)) return BigIntVal::null();
+ return {MaskShowLastNImpl(val, char_count.val, masked_number)};
+}
+BigIntVal MaskFunctions::MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return {MaskShowLastNImpl(val, char_count.val, number_char.val)};
+}
+
+/// MaskFirstN overloads for string value
+StringVal MaskFunctions::MaskFirstN(FunctionContext *ctx, const StringVal &val) {
+ return MaskFirstNImpl(ctx, val, CHAR_COUNT, MASKED_UPPERCASE, MASKED_LOWERCASE,
+ MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count) {
+ return MaskFirstNImpl(ctx, val, char_count.val, MASKED_UPPERCASE, MASKED_LOWERCASE,
+ MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char) {
+ return MaskFirstNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ GetFirstChar(other_char, MASKED_OTHER_CHAR));
+}
+StringVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return MaskFirstN(ctx, val, char_count, upper_char, lower_char, digit_char,
+ other_char);
+}
+StringVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ return MaskFirstNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ other_char.val);
+}
+StringVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return MaskFirstNImpl(ctx, val, char_count.val, upper_char.val, lower_char.val,
+ digit_char.val, other_char.val);
+}
+
+/// MaskFirstN overloads for numeric value
+BigIntVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const BigIntVal& val) {
+ return {MaskFirstNImpl(val, CHAR_COUNT, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count) {
+ return {MaskFirstNImpl(val, char_count.val, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return {MaskFirstNImpl(val, char_count.val, number_char.val)};
+}
+BigIntVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ int masked_number;
+ if (!GetDigitFromString(ctx, number_char, &masked_number)) return BigIntVal::null();
+ return {MaskFirstNImpl(val, char_count.val, masked_number)};
+}
+BigIntVal MaskFunctions::MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return {MaskFirstNImpl(val, char_count.val, number_char.val)};
+}
+
+/// MaskLastN overloads for string value
+StringVal MaskFunctions::MaskLastN(FunctionContext *ctx, const StringVal &val) {
+ return MaskLastNImpl(ctx, val, CHAR_COUNT, MASKED_UPPERCASE, MASKED_LOWERCASE,
+ MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count) {
+ return MaskLastNImpl(ctx, val, char_count.val, MASKED_UPPERCASE, MASKED_LOWERCASE,
+ MASKED_DIGIT, MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char) {
+ return MaskLastNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ GetFirstChar(other_char, MASKED_OTHER_CHAR));
+}
+StringVal MaskFunctions::MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return MaskLastN(ctx, val, char_count, upper_char, lower_char, digit_char,
+ other_char);
+}
+StringVal MaskFunctions::MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ return MaskLastNImpl(ctx, val, char_count.val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ other_char.val);
+}
+StringVal MaskFunctions::MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return MaskLastNImpl(ctx, val, char_count.val, upper_char.val, lower_char.val,
+ digit_char.val, other_char.val);
+}
+
+/// MaskLastN overloads for numeric value
+BigIntVal MaskFunctions::MaskLastN(FunctionContext* ctx, const BigIntVal& val) {
+ return {MaskLastNImpl(val, CHAR_COUNT, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count) {
+ return {MaskLastNImpl(val, char_count.val, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return {MaskLastNImpl(val, char_count.val, number_char.val)};
+}
+BigIntVal MaskFunctions::MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ int masked_number;
+ if (!GetDigitFromString(ctx, number_char, &masked_number)) return BigIntVal::null();
+ return {MaskLastNImpl(val, char_count.val, masked_number)};
+}
+BigIntVal MaskFunctions::MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char) {
+ return {MaskLastNImpl(val, char_count.val, number_char.val)};
+}
+
+/// Mask() overloads for string value
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val) {
+ return MaskImpl(ctx, val, MASKED_UPPERCASE, MASKED_LOWERCASE, MASKED_DIGIT,
+ MASKED_OTHER_CHAR);
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char) {
+ return MaskImpl(ctx, val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ GetFirstChar(other_char, MASKED_OTHER_CHAR));
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return Mask(ctx, val, upper_char, lower_char, digit_char, other_char);
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char) {
+ return MaskImpl(ctx, val, upper_char.val, lower_char.val, digit_char.val,
+ other_char.val);
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ return MaskImpl(ctx, val,
+ GetFirstChar(upper_char, MASKED_UPPERCASE),
+ GetFirstChar(lower_char, MASKED_LOWERCASE),
+ GetFirstChar(digit_char, MASKED_DIGIT),
+ other_char.val);
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value) {
+ return Mask(ctx, val, upper_char, lower_char, digit_char, other_char);
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value) {
+ return Mask(ctx, val, upper_char, lower_char, digit_char, other_char, number_char);
+}
+StringVal MaskFunctions::Mask(FunctionContext* ctx, const StringVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char, const IntVal& day_value,
+ const IntVal& month_value, const IntVal& year_value) {
+ return MaskImpl(ctx, val, upper_char.val, lower_char.val, digit_char.val,
+ other_char.val);
+}
+
+/// Mask() overloads for Date value
+DateVal MaskFunctions::Mask(FunctionContext* ctx, const DateVal& val) {
+ return MaskImpl(ctx, val, MASKED_DAY_COMPONENT_VAL, MASKED_MONTH_COMPONENT_VAL,
+ MASKED_YEAR_COMPONENT_VAL);
+}
+DateVal MaskFunctions::Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value) {
+ return MaskImpl(ctx, val, day_value.val, MASKED_MONTH_COMPONENT_VAL,
+ MASKED_YEAR_COMPONENT_VAL);
+}
+DateVal MaskFunctions::Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value) {
+ return MaskImpl(ctx, val, day_value.val, month_value.val, MASKED_YEAR_COMPONENT_VAL);
+}
+DateVal MaskFunctions::Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value) {
+ return MaskImpl(ctx, val, day_value.val, month_value.val, year_value.val);
+}
+DateVal MaskFunctions::Mask(FunctionContext* ctx, const DateVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char, const IntVal& day_value,
+ const IntVal& month_value, const IntVal& year_value) {
+ return MaskImpl(ctx, val, day_value.val, month_value.val, year_value.val);
+}
+DateVal MaskFunctions::Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value) {
+ return MaskImpl(ctx, val, day_value.val, month_value.val, year_value.val);
+}
+
+/// Mask() overloads for numeric value
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val) {
+ return {MaskShowFirstNImpl(val, 0, MASKED_NUMBER)};
+}
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char) {
+ return {MaskShowFirstNImpl(val, 0, number_char.val)};
+}
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char) {
+ int masked_number;
+ if (!GetDigitFromString(ctx, number_char, &masked_number)) return BigIntVal::null();
+ return {MaskShowFirstNImpl(val, 0, masked_number)};
+}
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char) {
+ return {MaskShowFirstNImpl(val, 0, number_char.val)};
+}
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value) {
+ return {MaskShowFirstNImpl(val, 0, number_char.val)};
+}
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value) {
+ int masked_number;
+ if (!GetDigitFromString(ctx, number_char, &masked_number)) return BigIntVal::null();
+ return {MaskShowFirstNImpl(val, 0, masked_number)};
+}
+BigIntVal MaskFunctions::Mask(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char, const IntVal& day_value,
+ const IntVal& month_value, const IntVal& year_value) {
+ return {MaskShowFirstNImpl(val, 0, number_char.val)};
+}
+
+StringVal MaskFunctions::MaskHash(FunctionContext* ctx, const StringVal& val) {
+ // Hive hash the value by sha256 and encoding it into a lower case hex string.
+ StringVal sha256_hash(ctx, SHA256_DIGEST_LENGTH);
+ if (UNLIKELY(sha256_hash.is_null)) return StringVal::null();
+ discard_result(SHA256(val.ptr, val.len, sha256_hash.ptr));
+ return StringFunctions::Lower(ctx, MathFunctions::HexString(ctx, sha256_hash));
+}
+// For other types, the hash values are always NULL.
+BigIntVal MaskFunctions::MaskHash(FunctionContext* ctx, const BigIntVal& val) {
+ return BigIntVal::null();
+}
+DoubleVal MaskFunctions::MaskHash(FunctionContext* ctx, const DoubleVal& val) {
+ return DoubleVal::null();
+}
+BooleanVal MaskFunctions::MaskHash(FunctionContext* ctx, const BooleanVal& val) {
+ return BooleanVal::null();
+}
+TimestampVal MaskFunctions::MaskHash(FunctionContext* ctx, const TimestampVal& val) {
+ return TimestampVal::null();
+}
+DateVal MaskFunctions::MaskHash(FunctionContext* ctx, const DateVal& val) {
+ return DateVal::null();
+}
diff --git a/be/src/exprs/mask-functions.h b/be/src/exprs/mask-functions.h
new file mode 100644
index 0000000..0d3d830
--- /dev/null
+++ b/be/src/exprs/mask-functions.h
@@ -0,0 +1,340 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "udf/udf.h"
+
+namespace impala {
+
+using impala_udf::FunctionContext;
+using impala_udf::IntVal;
+using impala_udf::BooleanVal;
+using impala_udf::BigIntVal;
+using impala_udf::DateVal;
+using impala_udf::DoubleVal;
+using impala_udf::TimestampVal;
+using impala_udf::StringVal;
+
+/// Data mask functions for Ranger column masking policies. In Hive, there're 6 builtin
+/// GenericUDFs for column masking:
+/// mask_show_first_n(value, charCount, upperChar, lowerChar, digitChar, otherChar,
+/// numberChar)
+/// mask_show_last_n(value, charCount, upperChar, lowerChar, digitChar, otherChar,
+/// numberChar)
+/// mask_first_n(value, charCount, upperChar, lowerChar, digitChar, otherChar,
+/// numberChar)
+/// mask_last_n(value, charCount, upperChar, lowerChar, digitChar, otherChar,
+/// numberChar)
+/// mask_hash(value)
+/// mask(value, upperChar, lowerChar, digitChar, otherChar, numberChar, dayValue,
+/// monthValue, yearValue)
+///
+/// Description of the parameters:
+/// value - value to mask. Supported types: TINYINT, SMALLINT, INT, BIGINT,
+/// STRING, VARCHAR, CHAR, DATE(only for mask()).
+/// charCount - number of characters. Default value: 4
+/// upperChar - character to replace upper-case characters with. Specify -1 to retain
+/// original character. Default value: 'X'
+/// lowerChar - character to replace lower-case characters with. Specify -1 to retain
+/// original character. Default value: 'x'
+/// digitChar - character to replace digit characters with. Specify -1 to retain
+/// original character. Default value: 'n'
+/// otherChar - character to replace all other characters with. Specify -1 to retain
+/// original character. Default value: -1
+/// numberChar - character to replace digits in a number with. Valid values: 0-9.
+/// Default value: '1'
+/// dayValue - value to replace day field in a date with. The day must be valid for
+/// the year and month, otherwise the results will be NULL. Specify -1 to
+/// retain original value. Valid values: 1-31. Default value: 1
+/// monthValue - value to replace month field in a date with. Specify -1 to retain
+/// original value. Valid values: 0-11. Default value: 0
+/// yearValue - value to replace year field in a date with. Specify -1 to retain
+/// original value. Default value: 1
+///
+/// In Hive, these functions accept variable length of arguments in non-restricted types:
+/// mask_show_first_n(val)
+/// mask_show_first_n(val, 8)
+/// mask_show_first_n(val, 8, 'X', 'x', 'n')
+/// mask_show_first_n(val, 8, 'x', 'x', 'x', 'x', -1)
+/// mask_show_first_n(val, 8, 'x', -1, 'x', 'x', '9')
+/// The arguments of upperChar, lowerChar, digitChar, otherChar and numberChar can be in
+/// string or numeric types.
+///
+/// We currently don't have a corresponding framework for GenericUDF(IMPALA-9271), so we
+/// implement these by overloads. However, it may requires hundreds of overloads to cover
+/// all possible combinations. We just implement some important overloads, including
+/// * those used by Ranger default masking policies,
+/// * those with simple arguments and may be useful for users,
+/// * an overload with all arguments in int type for full functionality. Char argument
+/// need to be converted to their ASCII value.
+class MaskFunctions {
+ public:
+ /// Declarations of mask_show_first_n()
+ /// Overloads for masking a string value
+ static StringVal MaskShowFirstN(FunctionContext* ctx, const StringVal& val);
+ static StringVal MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count);
+ static StringVal MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char);
+ static StringVal MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ // The default transformer of MASK_SHOW_FIRST_4 mask type is
+ // mask_show_first_n({col}, 4, 'x', 'x', 'x', -1, '1')
+ // So we need this overload.
+ static StringVal MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static StringVal MaskShowFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+ /// Overloads for masking a numeric value
+ static BigIntVal MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val);
+ static BigIntVal MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count);
+ static BigIntVal MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ static BigIntVal MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static BigIntVal MaskShowFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+
+ /// Declarations of mask_show_last_n()
+ /// Overloads for masking a string value
+ static StringVal MaskShowLastN(FunctionContext* ctx, const StringVal& val);
+ static StringVal MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count);
+ static StringVal MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char);
+ static StringVal MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ // The default transformer of MASK_SHOW_LAST_4 mask type is
+ // mask_show_last_n({col}, 4, 'x', 'x', 'x', -1, '1')
+ // So we need this overload.
+ static StringVal MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static StringVal MaskShowLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+ /// Overloads for masking a numeric value
+ static BigIntVal MaskShowLastN(FunctionContext* ctx, const BigIntVal& val);
+ static BigIntVal MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count);
+ static BigIntVal MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ static BigIntVal MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static BigIntVal MaskShowLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+
+ /// Declarations of mask_first_n()
+ /// Overloads for masking a string value
+ static StringVal MaskFirstN(FunctionContext* ctx, const StringVal& val);
+ static StringVal MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count);
+ static StringVal MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char);
+ static StringVal MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ // The default transformer of MASK_FIRST_4 mask type is
+ // mask_first_n({col}, 4, 'x', 'x', 'x', -1, '1')
+ // So we need this overload.
+ static StringVal MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static StringVal MaskFirstN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+ /// Overloads for masking a numeric value
+ static BigIntVal MaskFirstN(FunctionContext* ctx, const BigIntVal& val);
+ static BigIntVal MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count);
+ static BigIntVal MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ static BigIntVal MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static BigIntVal MaskFirstN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+
+ /// Declarations of mask_first_n()
+ /// Overloads for masking a string value
+ static StringVal MaskLastN(FunctionContext* ctx, const StringVal& val);
+ static StringVal MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count);
+ static StringVal MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char);
+ static StringVal MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ // The default transformer of MASK_SHOW_LAST_4 mask type is
+ // mask_show_last_n({col}, 4, 'x', 'x', 'x', -1, '1')
+ // So we need this overload.
+ static StringVal MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static StringVal MaskLastN(FunctionContext* ctx, const StringVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+ /// Overloads for masking a numeric value
+ static BigIntVal MaskLastN(FunctionContext* ctx, const BigIntVal& val);
+ static BigIntVal MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count);
+ static BigIntVal MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ static BigIntVal MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static BigIntVal MaskLastN(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& char_count, const IntVal& upper_char, const IntVal& lower_char,
+ const IntVal& digit_char, const IntVal& other_char, const IntVal& number_char);
+
+ /// Declarations of mask()
+ /// Overloads for masking a string value
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val);
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char);
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char);
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value);
+ // The default transformer of MASK_DATE_SHOW_YEAR mask type is
+ // mask({col}, 'x', 'x', 'x', -1, '1', 1, 0, -1)
+ // So we need this overload.
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value);
+ static StringVal Mask(FunctionContext* ctx, const StringVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char, const IntVal& day_value,
+ const IntVal& month_value, const IntVal& year_value);
+ /// Overloads for masking a date value
+ static DateVal Mask(FunctionContext* ctx, const DateVal& val);
+ static DateVal Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value);
+ static DateVal Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value);
+ // The default transformer of MASK_DATE_SHOW_YEAR mask type is
+ // mask({col}, 'x', 'x', 'x', -1, '1', 1, 0, -1)
+ // So we need this overload.
+ static DateVal Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value);
+ static DateVal Mask(FunctionContext* ctx, const DateVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value);
+ static DateVal Mask(FunctionContext* ctx, const DateVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char, const IntVal& day_value,
+ const IntVal& month_value, const IntVal& year_value);
+ /// Overloads for masking a numeric value
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val);
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char);
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char);
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char);
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const StringVal& other_char,
+ const IntVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value);
+ // The default transformer of MASK_DATE_SHOW_YEAR mask type is
+ // mask({col}, 'x', 'x', 'x', -1, '1', 1, 0, -1)
+ // So we need this overload.
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val,
+ const StringVal& upper_char, const StringVal& lower_char,
+ const StringVal& digit_char, const IntVal& other_char,
+ const StringVal& number_char, const IntVal& day_value, const IntVal& month_value,
+ const IntVal& year_value);
+ static BigIntVal Mask(FunctionContext* ctx, const BigIntVal& val,
+ const IntVal& upper_char, const IntVal& lower_char, const IntVal& digit_char,
+ const IntVal& other_char, const IntVal& number_char, const IntVal& day_value,
+ const IntVal& month_value, const IntVal& year_value);
+
+ /// Declarations of mask_hash()
+ static StringVal MaskHash(FunctionContext* ctx, const StringVal& val);
+ static BigIntVal MaskHash(FunctionContext* ctx, const BigIntVal& val);
+ static DoubleVal MaskHash(FunctionContext* ctx, const DoubleVal& val);
+ static BooleanVal MaskHash(FunctionContext* ctx, const BooleanVal& val);
+ static TimestampVal MaskHash(FunctionContext* ctx, const TimestampVal& val);
+ static DateVal MaskHash(FunctionContext* ctx, const DateVal& val);
+};
+
+} // namespace impala
diff --git a/be/src/exprs/scalar-expr-evaluator.cc b/be/src/exprs/scalar-expr-evaluator.cc
index 40723e7..57a4098 100644
--- a/be/src/exprs/scalar-expr-evaluator.cc
+++ b/be/src/exprs/scalar-expr-evaluator.cc
@@ -37,6 +37,7 @@
#include "exprs/is-null-predicate.h"
#include "exprs/like-predicate.h"
#include "exprs/literal.h"
+#include "exprs/mask-functions.h"
#include "exprs/math-functions.h"
#include "exprs/null-literal.h"
#include "exprs/operators.h"
@@ -436,6 +437,7 @@
IsNullPredicate::IsNull(nullptr, BooleanVal::null());
LikePredicate::Like(nullptr, StringVal::null(), StringVal::null());
Operators::Add_IntVal_IntVal(nullptr, IntVal::null(), IntVal::null());
+ MaskFunctions::MaskShowFirstN(nullptr, StringVal::null());
MathFunctions::Pi(nullptr);
StringFunctions::Length(nullptr, StringVal::null());
TimestampFunctions::Year(nullptr, TimestampVal::null());
diff --git a/common/function-registry/impala_functions.py b/common/function-registry/impala_functions.py
index f132743..2ea8597 100644
--- a/common/function-registry/impala_functions.py
+++ b/common/function-registry/impala_functions.py
@@ -810,6 +810,125 @@
[['shiftright'], 'SMALLINT', ['SMALLINT', 'INT'], 'impala::BitByteFunctions::ShiftRight'],
[['shiftright'], 'INT', ['INT', 'INT'], 'impala::BitByteFunctions::ShiftRight'],
[['shiftright'], 'BIGINT', ['BIGINT', 'INT'], 'impala::BitByteFunctions::ShiftRight'],
+
+ # Mask functions
+ [['mask_show_first_n'], 'STRING', ['STRING'], 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'STRING', ['STRING', 'INT'], 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'STRING', ['STRING', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'BIGINT', ['BIGINT'], 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'BIGINT', ['BIGINT', 'INT'], 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_first_n'], 'BIGINT', ['BIGINT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskShowFirstN'],
+ [['mask_show_last_n'], 'STRING', ['STRING'], 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'STRING', ['STRING', 'INT'], 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'STRING', ['STRING', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'BIGINT', ['BIGINT'], 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'BIGINT', ['BIGINT', 'INT'], 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_show_last_n'], 'BIGINT', ['BIGINT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskShowLastN'],
+ [['mask_first_n'], 'STRING', ['STRING'], 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'STRING', ['STRING', 'INT'], 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'STRING', ['STRING', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'BIGINT', ['BIGINT'], 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'BIGINT', ['BIGINT', 'INT'], 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_first_n'], 'BIGINT', ['BIGINT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskFirstN'],
+ [['mask_last_n'], 'STRING', ['STRING'], 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'STRING', ['STRING', 'INT'], 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'STRING', ['STRING', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'STRING', ['STRING', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'BIGINT', ['BIGINT'], 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'BIGINT', ['BIGINT', 'INT'], 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'BIGINT', ['BIGINT', 'INT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask_last_n'], 'BIGINT', ['BIGINT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::MaskLastN'],
+ [['mask'], 'STRING', ['STRING'], 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'STRING', 'STRING', 'STRING', 'STRING'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'STRING', 'STRING', 'STRING', 'STRING', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'STRING', 'STRING', 'STRING', 'INT', 'STRING', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'STRING', ['STRING', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT'], 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT', 'STRING', 'STRING', 'STRING', 'STRING', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT', 'STRING', 'STRING', 'STRING', 'INT', 'STRING', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'BIGINT', ['BIGINT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'DATE', ['DATE'], 'impala::MaskFunctions::Mask'],
+ [['mask'], 'DATE', ['DATE', 'STRING', 'STRING', 'STRING', 'STRING', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'DATE', ['DATE', 'STRING', 'STRING', 'STRING', 'STRING', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'DATE', ['DATE', 'STRING', 'STRING', 'STRING', 'INT', 'STRING', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'DATE', ['DATE', 'STRING', 'STRING', 'STRING', 'STRING', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask'], 'DATE', ['DATE', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT', 'INT'],
+ 'impala::MaskFunctions::Mask'],
+ [['mask_hash'], 'STRING', ['STRING'], 'impala::MaskFunctions::MaskHash'],
+ [['mask_hash'], 'BIGINT', ['BIGINT'], 'impala::MaskFunctions::MaskHash'],
+ [['mask_hash'], 'DOUBLE', ['DOUBLE'], 'impala::MaskFunctions::MaskHash'],
+ [['mask_hash'], 'BOOLEAN', ['BOOLEAN'], 'impala::MaskFunctions::MaskHash'],
+ [['mask_hash'], 'TIMESTAMP', ['TIMESTAMP'], 'impala::MaskFunctions::MaskHash'],
+ [['mask_hash'], 'DATE', ['DATE'], 'impala::MaskFunctions::MaskHash'],
]
invisible_functions = [
