Apache Falcon Changelog

Release 0.10 - 2016-07-26

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
FALCON-1926Filter out effectively non-falcon related JMS messages from OozieMajormessagingVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1858Support HBase as a storage backend for Falcon Titan graphDBMajor.Ying ZhengVenkat Ranganathan
FALCON-1852Optional Input for a process not truly optionalMajor.Pallavi RaoPallavi Rao
FALCON-1844Falcon feed replication leaves behind old files when a feed instance is re-runMajor.Pallavi RaoPallavi Rao
FALCON-1835Falcon should do coord rerun rather than workflow rerun to ensure concurrencyMajor.Pallavi RaoPallavi Rao

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
FALCON-1956Graphite Plugin for monitoringMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1919Provide user the option to store sensitive information with Hadoop credential providerMajor.Ying ZhengYing Zheng
FALCON-1865Persist Feed sla data to databaseMajor.Ajay YadavaPraveen Adlakha
FALCON-1861Support HDFS Snapshot based replication in FalconMajorreplicationBalu VellankiBalu Vellanki
FALCON-1763Create a spark execution engine for FalconMajor.Venkat RanganathanPeeyush Bishnoi
FALCON-1627Provider integration with Azure Data Factory pipelinesMajor.Venkat RanganathanYing Zheng
FALCON-1623Implement Safe Mode in FalconMajor.sandeep samudralaBalu Vellanki
FALCON-1333Support instance search of a group of entitiesMajor.Ying ZhengYing Zheng
FALCON-634Add server side extensions in FalconMajor.Venkatesh SeetharamSowmya Ramesh
FALCON-141Support cluster updatesMajor.Shwetha G SBalu Vellanki
FALCON-36Ability to ingest data from databasesMajoracquisitionVenkatesh SeetharamVenkatesan Ramachandran

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
FALCON-2020Changes in Graphite Notification PluginMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1981Remove runtime superfluous jar dependencies - pom.xml cleanupMajorbuild-toolsVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1979Update HttpClient versions to close security vulnerabilitiesMajor.Balu VellankiBalu Vellanki
FALCON-1963Falcon CLI should provide detailed hints if the user's command is invalidMajor.Ying ZhengYing Zheng
FALCON-1942Allow Falcon server and client classpath to be customizableMajor.Venkat RanganathanVenkat Ranganathan
FALCON-1916Allow RM principal to be specified in Cluster entityMajorcommonVenkat RanganathanVenkat Ranganathan
FALCON-1895Refactoring of FalconCLI and FalconClientMajorclientPraveen AdlakhaPraveen Adlakha
FALCON-1841Grouping test in falcon for running nightly regressionMajorregressionPragya MittalPragya Mittal
FALCON-1836Ingest to HiveMajor.Venkatesan RamachandranVenkatesan Ramachandran
FALCON-1802Workflow Builder for scheduling based on Data for Process in case of Native SchedulerMajor.pavan kumar kolamuripavan kumar kolamuri
FALCON-1774Better message for api not allowed on serverMajor.Sanjeev TPraveen Adlakha
FALCON-1751Support assembly:single mojoMinor.ruoyu wangruoyu wang
FALCON-887Support for multiple lib paths in falcon processMinorprocessAkshay GoyalSowmya Ramesh
FALCON-625Documentation improvementsMajor.Paul IsaychukAjay Yadava

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
FALCON-2107NPE in FalconWorkflowEngine::isActive() methodBlocker.Balu VellankiBalu Vellanki
FALCON-2104Loss of data in GraphDB when upgrading Falcon from 0.9 to 0.10Blocker.Balu VellankiBalu Vellanki
FALCON-2100Remove dependency on com.vividsolutions.jtsMajor.Balu VellankiBalu Vellanki
FALCON-2090HDFS Snapshot failed with UnknownHostException when scheduling in HA ModeCriticalreplicationMurali RamasamiBalu Vellanki
FALCON-2088Entity submission fails with EntityNotRegisteredException in distributed modeBlockerfeed, prism, processPragya MittalPraveen Adlakha
FALCON-2084HCatReplicationTest are failing in secure modeCriticalreplicationMurali RamasamiVenkat Ranganathan
FALCON-2081ExtensionManagerIT fails occassionallyBlockertestsBalu VellankiBalu Vellanki
FALCON-2076Server fails to start since extension.store.uri is not defined in startup.propertiesMajorprismPragya MittalBalu Vellanki
FALCON-2075Falcon HiveDR tasks do not report progress and can get killedCritical.Venkat RanganathanVenkat Ranganathan
FALCON-2071Falcon Spark SQL failing with Yarn Client ModeCriticalprocessMurali RamasamiPeeyush Bishnoi
FALCON-2061Falcon CLI shows hadoop classpath loading info in the consoleMajorclientMurali RamasamiBalu Vellanki
FALCON-2060Retry does not happen if instance timedoutMajor.Pragya MittalPallavi Rao
FALCON-2058s3 tests with dummy url no longer compatible with latest HDFSMajor.Ying ZhengYing Zheng
FALCON-2057HiveDR not working with multiple users and same DBMajorreplicationMurali RamasamiBalu Vellanki
FALCON-2056HiveDR doesn't work with multiple usersMajorreplicationMurali RamasamiSowmya Ramesh
FALCON-2051Falcon post-processing services are not getting invokedBlockergeneralPeeyush BishnoiVenkatesan Ramachandran
FALCON-2050Configure jetty parent classloader to be prioritized over webapp classloaderMajorcommonVenkat RanganathanVenkat Ranganathan
FALCON-2049Feed Replication with Empty Directories are failingBlockerfeedMurali RamasamiBalu Vellanki
FALCON-2048Cluster submission failed in yarn-cluster modeCriticalgeneralMurali RamasamiPeeyush Bishnoi
FALCON-2046HDFS Replication failing in secure ModeCriticalreplicationMurali RamasamiSowmya Ramesh
FALCON-2045Enhance document on registry point in cluster specifiction for Hive HA modeMajor.Ying ZhengYing Zheng
FALCON-2038When all Optional input instances are missing, we should not suffix partitionMajor.Pallavi RaoPallavi Rao
FALCON-2037HiveDR Extension tests are failed in Secure mode with clusterForJobNNKerberosPrincipal not foundCriticalreplicationMurali RamasamiSowmya Ramesh
FALCON-2036Update twiki on entity list operation with up-to-date REST API pathMajor.Ying ZhengYing Zheng
FALCON-2035Entity list operation without type parameter doesn't work when authorization is enabledMajor.Ying ZhengYing Zheng
FALCON-2034Make numThreads and timeOut configurable In ConfigurationStore initCritical.Pallavi Raosandeep samudrala
FALCON-2032Update the extension documentation to add ExtensionService before ConfigurationStore in startup propertiesMajor.Sowmya RameshSowmya Ramesh
FALCON-2031Hcat Retention test cases are failing with NoClassDefFoundErrorBlockerretentionPeeyush BishnoiPeeyush Bishnoi
FALCON-2027Enhance documentation on data replication from HDP to AzureMajor.Ying ZhengYing Zheng
FALCON-2025Periodic revalidation of kerberos credentials should be done on loginUserMajor.Balu VellankiBalu Vellanki
FALCON-2023Feed eviction fails when feed locations “stats” and “meta” does not have time pattern.BlockerfeedBalu VellankiVenkatesan Ramachandran
FALCON-2018WorkflowJobNotification sends incorrect message for killed instancesMajor.Pragya MittalPraveen Adlakha
FALCON-2017Fix HiveDR extension issuesMajor.Sowmya RameshSowmya Ramesh
FALCON-2016maven assembly:single fails on MacOSMajor.Pallavi RaoPallavi Rao
FALCON-2010Fix UT errors due to ActiveMQ upgradeMajor.Ying ZhengYing Zheng
FALCON-2007Hive DR Replication failing with “Can not create a Path from a null string”Critical.Peeyush BishnoiPeeyush Bishnoi
FALCON-1984Provide proper hint and documentation if required titan storage backend is not configured in startup.propertiesMajor.Ying ZhengYing Zheng
FALCON-1983Upgrade jackson core and databind versions to fix dependency incompatibility with higher-version HiveMajor.Ying ZhengYing Zheng
FALCON-1982Document use of HBase in standalone mode for GraphDBMajordocsVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1978Fix flaky unit test - MetadataMappingServiceTestMajortestsVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1976Remove hadoop-2 profileMajor.Venkat RanganathanVenkat Ranganathan
FALCON-1975Getting NoSuchMethodError when calling isNoneEmptyMajor.Ying ZhengYing Zheng
FALCON-1974Cluster update : Allow superuser to update bundle/coord of dependent entitiesMajor.Balu VellankiBalu Vellanki
FALCON-1973Falcon build failure due checkstyle issueMajor.Peeyush BishnoiPeeyush Bishnoi
FALCON-1972Handling cases when Extension service or “extension.store.uri” is not present in startup proeprtiesMajor.Sowmya RameshSowmya Ramesh
FALCON-1969Provide server-side error details on CLI, if anyMajor.Ying ZhengYing Zheng
FALCON-1965Update ActiveMQ version to 5.13.3 to avoid Falcon start error after rolling upgradeMajor.Ying ZhengYing Zheng
FALCON-1964Should delete temporary JKS file after IT tests for credential provider aliasMajor.Ying ZhengYing Zheng
FALCON-1962Extension related bugsMajor.Sowmya RameshSowmya Ramesh
FALCON-1961Should return error if an extension job doesn't exist for delete/suspend/resume/schedule operationsMajor.Ying ZhengYing Zheng
FALCON-1957Documentation on using Hadoop credential provider for sensitive propertiesMajor.Ying ZhengYing Zheng
FALCON-1953Build fails when profiles hivedr and test-patch is used togetherMajor.Balu VellankiBalu Vellanki
FALCON-1943Extension API/CLI fails when authorization is enabledMajor.Sowmya RameshSowmya Ramesh
FALCON-1941HiveDR fails with NN-HA enabled on both the source and target clustersCritical.Venkat RanganathanVenkat Ranganathan
FALCON-1939Avoid creating multiple falcon*.tar.gz during falcon buildMajorbuild-toolsBalu VellankiBalu Vellanki
FALCON-1936Extensions related files are not available in $FALCON_HOM/extensions/ directoryCritical.Peeyush BishnoiSowmya Ramesh
FALCON-1935Falcon fails to start with default startup.propertiesBlocker.Ying ZhengPraveen Adlakha
FALCON-1934Document safemode in Falcon ServerMajordocsBalu VellankiBalu Vellanki
FALCON-1932Extension CLI should support common optionsMajor.Ying ZhengYing Zheng
FALCON-1931multiCluster tag is missing for Multiple Cluster scenariosMajorregressionMurali RamasamiMurali Ramasami
FALCON-1928FalconJPAService missing in default startup.propertiesMajor.Pallavi RaoPallavi Rao
FALCON-1924Falcon Coordinator rerun return old workflow idMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1909Catalog instance triage action fails with null pointer exception.MajorfeedBalu VellankiBalu Vellanki
FALCON-1908Document HDFS snapshot based mirroring extensionMajor.Balu VellankiBalu Vellanki
FALCON-1907Package new CLI module addedMajorclientSowmya RameshSowmya Ramesh
FALCON-1896Failure in Falcon build in distro moduleMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1894HDFS Data replication cannot be initiated independent of Oozie server locationMinorgeneralAlex BushSowmya Ramesh
FALCON-1886Feed sla monitoring does not work across restartsMajor.Ajay YadavaAjay Yadava
FALCON-1885SLA monitoring API throws ResultNotFoundExceptionMajorfeedPragya MittalPraveen Adlakha
FALCON-1883Falcon regression build fails with minor checkstyle issuesMajorregressionMurali RamasamiMurali Ramasami
FALCON-1882Instance status api not working via prismMajorprismPragya MittalPraveen Adlakha
FALCON-1881Database Export should not expect fields list in the feed entity specificationMajoracquisitionVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1880To support TDE encryption : Add --skipcrccheck to distcp options for HiveDRMajorreplicationBalu VellankiBalu Vellanki
FALCON-1877Falcon webUI returns 413 (Full head - Request entity too large) error when TLS is enabled in a secure cluster with AD integrationMajor.Venkat RanganathanVenkat Ranganathan
FALCON-1874Import and Export fails with HDFS as src/destMajor.Pallavi RaoPallavi Rao
FALCON-1867hardcoded query names in JDBCStateStoreMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1866Bug in JDBCStateStoreMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1864Retry event does not get removed from delay queue even after the instance succeedsMajorrerunPallavi RaoPallavi Rao
FALCON-1859Database Export instances are not added graph db for lineage trackingMajorgeneralVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1855Falcon regression build fails with checkstyle issuesMajorregressionPragya MittalMurali Ramasami
FALCON-1854Fixing PrismProcessScheduleTest and NoOutputProcessTestMajorregressionMurali RamasamiMurali Ramasami
FALCON-1848Late rerun is not working due to failnodes set to trueMajorrerunPragya MittalPallavi Rao
FALCON-1847Execution order not honored when instances are suspended/resumedMajorschedulerPallavi RaoPallavi Rao
FALCON-1846Fixing EntityDryRunTestMajorregressionPragya MittalPragya Mittal
FALCON-1845Retries Stopped happening for all entities when one entity was deleted during rerun of instanceMajorrerunpavan kumar kolamuripavan kumar kolamuri
FALCON-1842Falcon build failed in Jenkins at org.apache.falcon.oozie.feed.OozieFeedWorkflowBuilderTestMajorfalcon-unitBalu VellankiBalu Vellanki
FALCON-1840Archive older definition in case of updateMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1838Export instances are not added graph db for lineage trackingMajor.Venkatesan RamachandranVenkatesan Ramachandran
FALCON-1826Execution order not honoured when instances are KILLEDMajorschedulerPragya MittalPallavi Rao
FALCON-1825Process end time inclusive in case of Native SchedulerMajorschedulerpavan kumar kolamuripavan kumar kolamuri
FALCON-1823wrong permissions on hadoolibs and conf folder in distributed mode debMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1819Improve test class entity cleanup logicMajormerlinPaul IsaychukPaul Isaychuk
FALCON-1816Fix findbugs-exclude.xml path and hadoop version in falcon-regression pomMajormerlinPaul IsaychukPaul Isaychuk
FALCON-1811Status API does not honour start optionMajorclientPragya MittalPraveen Adlakha
FALCON-1796[HOTFIX] Incorrect parent pom in distro moduleMajor.Ajay YadavaAjay Yadava
FALCON-1795Kill api does not kill waiting/ready instancesMajorooziePragya Mittalsandeep samudrala
FALCON-1793feed element action=“archive” is submittable via command line tool falconMajorfeedMargus RooDeepak Barr
FALCON-1792Upgrade hadoop.version to 2.6.2MajorhadoopVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1787Ooozie pig-action.xml requires hive sharelib for HCatalog useMajoroozieMark GreeneSowmya Ramesh
FALCON-1784Add regression test for for FALCON-1647MajormerlinPaul IsaychukPaul Isaychuk
FALCON-1783Fix ProcessUpdateTest and SearchApiTest to use prismMajormerlinPaul IsaychukPaul Isaychuk
FALCON-1766Add CLI metrics check for HiveDR, HDFS and feed replicationMajormerlinPaul IsaychukPaul Isaychuk
FALCON-1743Entity summary does not work via prismMajorclientPragya MittalAjay Yadava
FALCON-1724Falcon CLI.twiki in docs folder is not pointed by index pageMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1721Move checkstyle artifacts under parentMajor.Shwetha G Ssandeep samudrala
FALCON-1621Lifecycle of entity gets missed when prism and falcon server communicatesMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1584Falcon allows invalid hadoop queue name for schedulable feed entitiesMajor.Venkatesan RamachandranVenkatesan Ramachandran

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
FALCON-2099Update Installation-steps.txt and NOTICE.txt for 0.10 releaseMajoreaseBalu VellankiBalu Vellanki
FALCON-2000Create branch 0.10MajorgeneralBalu VellankiBalu Vellanki
FALCON-1996Upgrade falcon POM for 0.10 releaseMajorbuild-toolsBalu VellankiBalu Vellanki
FALCON-1993Update JIRA fix versionsMajorgeneralBalu VellankiBalu Vellanki
FALCON-1980Change input and output argument order for Spark process workflowMajor.Peeyush BishnoiPeeyush Bishnoi
FALCON-1954Steps to configure Oozie JMS for FalconMajormessagingVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1938Add support to execute Spark SQL processMajor.Peeyush BishnoiPeeyush Bishnoi
FALCON-1937Add documentation for cluster update.Major.Balu VellankiBalu Vellanki
FALCON-1929Extension job management: IT tests for CLIsMajor.Ying ZhengYing Zheng
FALCON-1905Extension Job Management: IT tests for REST APIs and CLIsMajor.Ying ZhengYing Zheng
FALCON-1904Extension Job Management: documentation for REST APIs and CLIMajor.Ying ZhengYing Zheng
FALCON-1902Server side extension repository management CLI supportMajor.Sowmya RameshSowmya Ramesh
FALCON-1897Extension Job Management: CLI supportMajor.Ying ZhengYing Zheng
FALCON-1893Add documentation and examples for spark workflow engineMajor.Peeyush BishnoiPeeyush Bishnoi
FALCON-1892Remove client side Recipe logicMajor.Sowmya RameshSowmya Ramesh
FALCON-1860ADFProviderService should be optional as default settingMajor.Ying ZhengYing Zheng
FALCON-1853Add spark process workflow builderMajorooziePeeyush BishnoiPeeyush Bishnoi
FALCON-1839Test case for APIs for entities scheduled on native schedulerMajorschedulerPragya MittalPragya Mittal
FALCON-1831Flaky WorkflowExecutionContextTest.testWorkflowStartEndMajor.Pallavi RaoPallavi Rao
FALCON-1829Add regression for submit and schedule process on native scheduler (time based)MajorschedulerPragya MittalPragya Mittal
FALCON-1817Update xsd for Spark execution engineMajor.Peeyush BishnoiPeeyush Bishnoi
FALCON-1801Update CHANGES.txt in trunk to mark 0.9 as releasedMajor.Pallavi RaoPallavi Rao
FALCON-1790CLI support for instance searchMajor.Ying ZhengYing Zheng
FALCON-1789Extension Job Management: REST APIMajor.Sowmya RameshYing Zheng
FALCON-1767Improve Falcon retention policy documentationMajor.Sowmya RameshSowmya Ramesh
FALCON-1729Database ingest to support password alias via keystore fileMajoracquisitionVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1646Ability to export to database - Entity DefinitionMajoracquisitionVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1496Flaky FalconPostProcessingTestMajor.Pallavi RaoPallavi Rao
FALCON-1335Backend support of instance search of a group of entitiesMajor.Ying ZhengYing Zheng
FALCON-1334Improve search performance with Titan graph database indexingMajor.Ying ZhengYing Zheng
FALCON-1111Instance update on titan DB based on JMS notifications on workflow jobsMajorcommon, messagingSowmya RameshYing Zheng
FALCON-1107Move trusted recipe processing to server sideMajor.Sowmya RameshSowmya Ramesh
FALCON-1106Documentation for extensionMajor.Sowmya RameshSowmya Ramesh
FALCON-1105Server side extension repository management REST API supportMajorclientSowmya RameshSowmya Ramesh
FALCON-1085Allow cluster entities to be updatedMajor.Ajay YadavaBalu Vellanki

OTHER:

JIRASummaryPriorityComponentReporterContributor
FALCON-2063Add change log for 0.10Major.Ajay YadavaAjay Yadava
FALCON-1765Move to github pull request modelMajor.Ajay YadavaAjay Yadava
FALCON-2008Add documentation for Graphite Notification PluginMajor.Praveen AdlakhaPraveen Adlakha
FALCON-1948Document steps to configure Oozie for FalconMajordocsVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1899Create examples artifact module in FalconMajor.Peeyush BishnoiPeeyush Bishnoi
FALCON-1888Falcon JMS Notification details and documentationMajordocsVenkatesan RamachandranVenkatesan Ramachandran
FALCON-1818Minor doc update for tar package locations after FALCON-1751Minor.Deepak BarrDeepak Barr
FALCON-1806Update documentation for Import and ExportMajor.Venkatesan RamachandranVenkatesan Ramachandran
FALCON-1567Test case for Lifecycle featureMajormerlinPragya MittalPragya Mittal
FALCON-1566Add test for SLA monitoring APIMajormerlinPragya MittalPragya Mittal