| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| |
| |
| <title>Apache Jena - Apache Jena Elephas - Map/Reduce API</title> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <link href="/css/bootstrap.min.css" rel="stylesheet" media="screen"> |
| <link href="/css/bootstrap-icons.css" rel="stylesheet" media="screen"><link rel="stylesheet" type="text/css" href="https://jena.apache.org/sass/jena.1b17c39a117e22b46db4c66f6395dc27c134a60377d87d2d5745b8600eb69722.css" integrity="sha256-GxfDmhF+IrRttMZvY5XcJ8E0pgN32H0tV0W4YA62lyI="> |
| <link rel="shortcut icon" href="/images/favicon.ico" /> |
| |
| </head> |
| |
| <body> |
| |
| <nav class="navbar navbar-expand-lg bg-body-tertiary" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| <a class="navbar-brand" href="/index.html"> |
| <img class="logo-menu" src="/images/jena-logo/jena-logo-notext-small.png" alt="jena logo">Apache Jena</a> |
| </div> |
| |
| <div class="collapse navbar-collapse" id="navbarNav"> |
| <ul class="navbar-nav me-auto mb-2 mb-lg-0"> |
| <li id="homepage" class="nav-item"><a class="nav-link" href="/index.html"><span class="bi-house"></span> Home</a></li> |
| <li id="download" class="nav-item"><a class="nav-link" href="/download/index.cgi"><span class="bi-download"></span> Download</a></li> |
| <li class="nav-item dropdown"> |
| <a href="#" class="nav-link dropdown-toggle" role="button" data-bs-toggle="dropdown" aria-expanded="false"><span class="bi-journal"></span> Learn <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li class="dropdown-header">Tutorials</li> |
| <li><a class="dropdown-item" href="/tutorials/index.html">Overview</a></li> |
| <li><a class="dropdown-item" href="/documentation/fuseki2/index.html">Fuseki Triplestore</a></li> |
| <li><a class="dropdown-item" href="/documentation/notes/index.html">How-To's</a></li> |
| <li><a class="dropdown-item" href="/documentation/query/manipulating_sparql_using_arq.html">Manipulating SPARQL using ARQ</a></li> |
| <li><a class="dropdown-item" href="/tutorials/rdf_api.html">RDF core API tutorial</a></li> |
| <li><a class="dropdown-item" href="/tutorials/sparql.html">SPARQL tutorial</a></li> |
| <li><a class="dropdown-item" href="/tutorials/using_jena_with_eclipse.html">Using Jena with Eclipse</a></li> |
| <li class="dropdown-divider"></li> |
| <li class="dropdown-header">References</li> |
| <li><a class="dropdown-item" href="/documentation/index.html">Overview</a></li> |
| <li><a class="dropdown-item" href="/documentation/query/index.html">ARQ (SPARQL)</a></li> |
| <li><a class="dropdown-item" href="/documentation/io/">RDF I/O</a></li> |
| <li><a class="dropdown-item" href="/documentation/assembler/index.html">Assembler</a></li> |
| <li><a class="dropdown-item" href="/documentation/tools/index.html">Command-line tools</a></li> |
| <li><a class="dropdown-item" href="/documentation/rdfs/">Data with RDFS Inferencing</a></li> |
| <li><a class="dropdown-item" href="/documentation/geosparql/index.html">GeoSPARQL</a></li> |
| <li><a class="dropdown-item" href="/documentation/inference/index.html">Inference API</a></li> |
| <li><a class="dropdown-item" href="/documentation/ontology/">Ontology API</a></li> |
| <li><a class="dropdown-item" href="/documentation/permissions/index.html">Permissions</a></li> |
| <li><a class="dropdown-item" href="/documentation/extras/querybuilder/index.html">Query Builder</a></li> |
| <li><a class="dropdown-item" href="/documentation/rdf/index.html">RDF API</a></li> |
| <li><a class="dropdown-item" href="/documentation/rdfconnection/">RDF Connection - SPARQL API</a></li> |
| <li><a class="dropdown-item" href="/documentation/rdfstar/index.html">RDF-star</a></li> |
| <li><a class="dropdown-item" href="/documentation/shacl/index.html">SHACL</a></li> |
| <li><a class="dropdown-item" href="/documentation/shex/index.html">ShEx</a></li> |
| <li><a class="dropdown-item" href="/documentation/tdb/index.html">TDB</a></li> |
| <li><a class="dropdown-item" href="/documentation/tdb2/index.html">TDB2</a></li> |
| <li><a class="dropdown-item" href="/documentation/query/text-query.html">Text Search</a></li> |
| </ul> |
| </li> |
| |
| <li class="nav-item dropdown"> |
| <a href="#" class="nav-link dropdown-toggle" role="button" data-bs-toggle="dropdown" aria-expanded="false"><span class="bi-journal-code"></span> Javadoc <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/documentation/javadoc.html">All Javadoc</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/arq/">ARQ</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/fuseki2/">Fuseki</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/geosparql/">GeoSPARQL</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/jena/">Jena Core</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/permissions/">Permissions</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/extras/querybuilder/">Query Builder</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/shacl/">SHACL</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/tdb/">TDB</a></li> |
| <li><a class="dropdown-item" href="/documentation/javadoc/text/">Text Search</a></li> |
| </ul> |
| </li> |
| </ul> |
| <form class="d-flex" role="search" action="/search" method="GET"> |
| <div class="input-group"> |
| <input class="form-control border-end-0 border m-0" type="search" name="q" id="search-query" placeholder="Search...." aria-label="Search" style="width: 10rem;"> |
| <button class="btn btn-outline-secondary border-start-0 border" type="submit"> |
| <i class="bi-search"></i> |
| </button> |
| </div> |
| </form> |
| <ul class="navbar-nav"> |
| <li id="ask" class="nav-item"><a class="nav-link" href="/help_and_support/index.html" title="Ask"><span class="bi-patch-question"></span><span class="text-body d-none d-xxl-inline"> Ask</span></a></li> |
| |
| <li class="nav-item dropdown"> |
| <a href="#" title="Get involved" class="nav-link dropdown-toggle" role="button" data-bs-toggle="dropdown" aria-expanded="false"><span class="bi-megaphone"></span><span class="text-body d-none d-xxl-inline"> Get involved </span><b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/getting_involved/index.html">Contribute</a></li> |
| <li><a class="dropdown-item" href="/help_and_support/bugs_and_suggestions.html">Report a bug</a></li> |
| <li class="dropdown-divider"></li> |
| <li class="dropdown-header">Project</li> |
| <li><a class="dropdown-item" href="/about_jena/about.html">About Jena</a></li> |
| <li><a class="dropdown-item" href="/about_jena/architecture.html">Architecture</a></li> |
| <li><a class="dropdown-item" href="/about_jena/citing.html">Citing</a></li> |
| <li><a class="dropdown-item" href="/about_jena/team.html">Project team</a></li> |
| <li><a class="dropdown-item" href="/about_jena/contributions.html">Related projects</a></li> |
| <li><a class="dropdown-item" href="/about_jena/roadmap.html">Roadmap</a></li> |
| <li><a class="dropdown-item" href="/about_jena/security-advisories.html">Security Advisories</a></li> |
| <li class="dropdown-divider"></li> |
| <li class="dropdown-header">ASF</li> |
| <li><a class="dropdown-item" href="https://www.apache.org/">Apache Software Foundation</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/foundation/sponsorship.html">Become a Sponsor</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/licenses/LICENSE-2.0">License</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| </ul> |
| </li> |
| |
| |
| |
| |
| <li class="nav-item" id="edit"><a class="nav-link" href="https://github.com/apache/jena-site/edit/main/source/documentation/archive/hadoop/mapred.md" title="Edit this page on GitHub"><span class="bi-pencil-square"></span><span class="text-body d-none d-xxl-inline"> Edit this page</span></a></li> |
| </ul> |
| </div> |
| </div> |
| </nav> |
| |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-12"> |
| |
| <div id="breadcrumbs"> |
|
|
|
|
|
|
|
|
|
|
|
|
| <ol class="breadcrumb mt-4 p-2 bg-body-tertiary">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| <li class="breadcrumb-item"><a href='/documentation'>DOCUMENTATION</a></li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| <li class="breadcrumb-item"><a href='/documentation/archive'>ARCHIVE</a></li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| <li class="breadcrumb-item"><a href='/documentation/archive/hadoop'>HADOOP</a></li>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| <li class="breadcrumb-item active">MAPRED.HTML</li>
|
|
|
|
|
|
|
|
|
| </ol>
|
|
|
|
|
|
|
| |
| </div> |
| <h1 class="title">Apache Jena Elephas - Map/Reduce API</h1> |
| |
| |
| <main class="d-flex flex-xl-row flex-column"> |
| |
| <aside class="text-muted align-self-start mb-3 p-0 d-xl-none d-block"> |
| <h2 class="h6 sticky-top m-0 p-2 bg-body-tertiary">On this page</h2> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#counting">Counting</a> |
| <ul> |
| <li><a href="#node-usage">Node Usage</a></li> |
| <li><a href="#literal-data-types">Literal Data Types</a></li> |
| <li><a href="#namespaces">Namespaces</a></li> |
| </ul> |
| </li> |
| <li><a href="#filtering">Filtering</a> |
| <ul> |
| <li><a href="#valid-data">Valid Data</a></li> |
| <li><a href="#ground-data">Ground Data</a></li> |
| <li><a href="#data-with-a-specific-uri">Data with a specific URI</a></li> |
| </ul> |
| </li> |
| <li><a href="#grouping">Grouping</a></li> |
| <li><a href="#splitting">Splitting</a></li> |
| <li><a href="#transforming">Transforming</a></li> |
| </ul> |
| |
| <ul> |
| <li><a href="#node-count">Node Count</a></li> |
| </ul> |
| </nav> |
| </aside> |
| <article class="flex-column me-lg-4"> |
| <p>The Map/Reduce API provides a range of building block <code>Mapper</code> and <code>Reducer</code> implementations that can be used as a starting point for building Map/Reduce applications that process RDF. Typically more complex applications will need to implement their own variants but these basic ones may still prove useful as part of a larger pipeline.</p> |
| <p>{% toc %}</p> |
| <h1 id="tasks">Tasks</h1> |
| <p>The API is divided based upon implementations that support various common Hadoop tasks with appropriate <code>Mapper</code> and <code>Reducer</code> implementations provided for each. In most cases these are implemented to be at least partially abstract to make it easy to implement customised versions of these.</p> |
| <p>The following common tasks are supported:</p> |
| <ul> |
| <li>Counting</li> |
| <li>Filtering</li> |
| <li>Grouping</li> |
| <li>Splitting</li> |
| <li>Transforming</li> |
| </ul> |
| <p>Note that standard Map/Reduce programming rules apply as normal. For example if a mapper/reducer transforms between data types then you need to make <code>setMapOutputKeyClass()</code>, <code>setMapOutputValueClass()</code>, <code>setOutputKeyClass()</code> and <code>setOutputValueClass()</code> calls on your Job configuration as necessary.</p> |
| <h2 id="counting">Counting</h2> |
| <p>Counting is one of the classic Map/Reduce tasks and features as both the official Map/Reduce example for both Hadoop itself and for Elephas. Implementations cover a number of different counting tasks that you might want to carry out upon RDF data, in most cases you will use the desired <code>Mapper</code> implementation in conjunction with the <code>NodeCountReducer</code>.</p> |
| <h3 id="node-usage">Node Usage</h3> |
| <p>The simplest type of counting supported is to count the usages of individual RDF nodes within the triples/quads. Depending on whether your data is triples/quads you can use either the <code>TripleNodeCountMapper</code> or the <code>QuadNodeCountMapper</code>.</p> |
| <p>If you want to count only usages of RDF nodes in a specific position then we also provide variants for that, for example <code>TripleSubjectCountMapper</code> counts only RDF nodes present in the subject position. You can substitute <code>Predicate</code> or <code>Object</code> into the class name in place of <code>Subject</code> if you prefer to count just RDF nodes in the predicate/object position instead. Similarly replace <code>Triple</code> with <code>Quad</code> if you wish to count usage of RDF nodes in specific positions of quads, an additional <code>QuadGraphCountMapper</code> if you want to calculate the size of graphs.</p> |
| <h3 id="literal-data-types">Literal Data Types</h3> |
| <p>Another interesting variant of counting is to count the usage of literal data types, you can use the <code>TripleDataTypeCountMapper</code> or <code>QuadDataTypeCountMapper</code> if you want to do this.</p> |
| <h3 id="namespaces">Namespaces</h3> |
| <p>Finally you may be interested in the usage of namespaces within your data, in this case the <code>TripleNamespaceCountMapper</code> or <code>QuadNamespaceCountMapper</code> can be used to do this. For this use case you should use the <code>TextCountReducer</code> to total up the counts for each namespace. Note that the mappers determine the namespace for a URI simply by splitting after the last <code>#</code> or <code>/</code> in the URI, if no such character exists then the full URI is considered to be the namespace.</p> |
| <h2 id="filtering">Filtering</h2> |
| <p>Filtering is another classic Map/Reduce use case, here you want to take the data and extract only the portions that you are interested in based on some criteria. All our filter <code>Mapper</code> implementations also support a Job configuration option named <code>rdf.mapreduce.filter.invert</code> allowing their effects to be inverted if desired e.g.</p> |
| <pre><code>config.setBoolean(RdfMapReduceConstants.FILTER_INVERT, true); |
| </code></pre> |
| <h3 id="valid-data">Valid Data</h3> |
| <p>One type of filter that may be useful particularly if you are generating RDF data that may not be strict RDF is the <code>ValidTripleFilterMapper</code> and the <code>ValidQuadFilterMapper</code>. These filters only keep triples/quads that are valid according to strict RDF semantics i.e.</p> |
| <ul> |
| <li>Subject can only be URI/Blank Node</li> |
| <li>Predicate can only be a URI</li> |
| <li>Object can be a URI/Blank Node/Literal</li> |
| <li>Graph can only be a URI or Blank Node</li> |
| </ul> |
| <p>If you wanted to extract only the bad data e.g. for debugging then you can of course invert these filters by setting <code>rdf.mapreduce.filter.invert</code> to <code>true</code> as shown above.</p> |
| <h3 id="ground-data">Ground Data</h3> |
| <p>In some cases you may only be interesting in triples/quads that are grounded i.e. don’t contain blank nodes in which case the <code>GroundTripleFilterMapper</code> and <code>GroundQuadFilterMapper</code> can be used.</p> |
| <h3 id="data-with-a-specific-uri">Data with a specific URI</h3> |
| <p>In lots of case you may want to extract only data where a specific URI occurs in a specific position, for example if you wanted to extract all the <code>rdf:type</code> declarations then you might want to use the <code>TripleFilterByPredicateUriMapper</code> or <code>QuadFilterByPredicateUriMapper</code> as appropriate. The job configuration option <code>rdf.mapreduce.filter.predicate.uris</code> is used to provide a comma separated list of the full URIs you want the filter to accept e.g.</p> |
| <pre><code>config.set(RdfMapReduceConstants.FILTER_PREDICATE_URIS, "http://example.org/predicate,http://another.org/predicate"); |
| </code></pre> |
| <p>Similar to the counting of node usage you can substitute <code>Predicate</code> for <code>Subject</code>, <code>Object</code> or <code>Graph</code> as desired. You will also need to do this in the job configuration option, for example to filter on subject URIs in quads use the <code>QuadFilterBySubjectUriMapper</code> and the <code>rdf.mapreduce.filter.subject.uris</code> configuration option e.g.</p> |
| <pre><code>config.set(RdfMapReduceConstants.FILTER_SUBJECT_URIS, "http://example.org/myInstance"); |
| </code></pre> |
| <h2 id="grouping">Grouping</h2> |
| <p>Grouping is again another frequent Map/Reduce use case, here we provide implementations that allow you to group triples or quads by a specific RDF node within the triples/quads e.g. by subject. For example to group quads by predicate use the <code>QuadGroupByPredicateMapper</code>, similar to filtering and counting you can substitute <code>Predicate</code> for <code>Subject</code>, <code>Object</code> or <code>Graph</code> if you wish to group by another node of the triple/quad.</p> |
| <h2 id="splitting">Splitting</h2> |
| <p>Splitting allows you to split triples/quads up into the constituent RDF nodes, we provide two kinds of splitting:</p> |
| <ul> |
| <li>To Nodes - Splits pairs of arbitrary keys with triple/quad values into several pairs of the key with the nodes as the values</li> |
| <li>With Nodes - Splits pairs of arbitrary keys with triple/quad values keeping the triple/quad as the key and the nodes as the values.</li> |
| </ul> |
| <h2 id="transforming">Transforming</h2> |
| <p>Transforming provides some very simple implementations that allow you to convert between triples and quads. For the lossy case of going from quads to triples simply use the <code>QuadsToTriplesMapper</code>.</p> |
| <p>If you want to go the other way - triples to quads - this requires adding a graph field to each triple and we provide two implementations that do that. Firstly there is <code>TriplesToQuadsBySubjectMapper</code> which puts each triple into a graph based on its subject i.e. all triples with a common subject go into a graph named for the subject. Secondly there is <code>TriplesToQuadsConstantGraphMapper</code> which simply puts all triples into the default graph, if you wish to change the target graph you should extend this class. If you wanted to select the graph to use based on some arbitrary criteria you should look at extending the <code>AbstractTriplesToQuadsMapper</code> instead.</p> |
| <h1 id="example-jobs">Example Jobs</h1> |
| <h2 id="node-count">Node Count</h2> |
| <p>The following example shows how to configure a job which performs a node count i.e. counts the usages of RDF terms (aka nodes in Jena parlance) within the data:</p> |
| <pre><code>// Assumes we have already created a Hadoop Configuration |
| // and stored it in the variable config |
| Job job = Job.getInstance(config); |
| |
| // This is necessary as otherwise Hadoop won't ship the JAR to all |
| // nodes and you'll get ClassDefNotFound and similar errors |
| job.setJarByClass(Example.class); |
| |
| // Give our job a friendly name |
| job.setJobName("RDF Triples Node Usage Count"); |
| |
| // Mapper class |
| // Since the output type is different from the input type have to declare |
| // our output types |
| job.setMapperClass(TripleNodeCountMapper.class); |
| job.setMapOutputKeyClass(NodeWritable.class); |
| job.setMapOutputValueClass(LongWritable.class); |
| |
| // Reducer class |
| job.setReducerClass(NodeCountReducer.class); |
| |
| // Input |
| // TriplesInputFormat accepts any RDF triples serialisation |
| job.setInputFormatClass(TriplesInputFormat.class); |
| |
| // Output |
| // NTriplesNodeOutputFormat produces lines consisting of a Node formatted |
| // according to the NTriples spec and the value separated by a tab |
| job.setOutputFormatClass(NTriplesNodeOutputFormat.class); |
| |
| // Set your input and output paths |
| FileInputFormat.setInputPath(job, new Path("/example/input")); |
| FileOutputFormat.setOutputPath(job, new Path("/example/output")); |
| |
| // Now run the job...</code></pre> |
| |
| </article> |
| |
| <aside class="text-muted align-self-start mb-3 mb-xl-5 p-0 d-none d-xl-flex flex-column sticky-top"> |
| <h2 class="h6 sticky-top m-0 p-2 bg-body-tertiary">On this page</h2> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#counting">Counting</a> |
| <ul> |
| <li><a href="#node-usage">Node Usage</a></li> |
| <li><a href="#literal-data-types">Literal Data Types</a></li> |
| <li><a href="#namespaces">Namespaces</a></li> |
| </ul> |
| </li> |
| <li><a href="#filtering">Filtering</a> |
| <ul> |
| <li><a href="#valid-data">Valid Data</a></li> |
| <li><a href="#ground-data">Ground Data</a></li> |
| <li><a href="#data-with-a-specific-uri">Data with a specific URI</a></li> |
| </ul> |
| </li> |
| <li><a href="#grouping">Grouping</a></li> |
| <li><a href="#splitting">Splitting</a></li> |
| <li><a href="#transforming">Transforming</a></li> |
| </ul> |
| |
| <ul> |
| <li><a href="#node-count">Node Count</a></li> |
| </ul> |
| </nav> |
| </aside> |
| </main> |
| |
| </div> |
| </div> |
| </div> |
| |
| <footer class="bd-footer py-4 py-md-5 mt-4 mt-lg-5 bg-body-tertiary"> |
| <div class="container" style="font-size:80%" > |
| <p> |
| Copyright © 2011–2024 The Apache Software Foundation, Licensed under the |
| <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>. |
| </p> |
| <p> |
| Apache Jena, Jena, the Apache Jena project logo, Apache and the Apache feather logos are trademarks of |
| The Apache Software Foundation. |
| <br/> |
| <a href="https://privacy.apache.org/policies/privacy-policy-public.html" |
| >Apache Software Foundation Privacy Policy</a>. |
| </p> |
| </div> |
| </footer> |
| |
| <script src="/js/popper.min.js.js" type="text/javascript"></script> |
| <script src="/js/bootstrap.min.js" type="text/javascript"></script> |
| <script src="/js/improve.js" type="text/javascript"></script> |
| |
| <script type="text/javascript"> |
| (function() { |
| 'use strict' |
| |
| |
| |
| const links = document.querySelectorAll(`a[href="${window.location.pathname}"]`) |
| if (links !== undefined && links !== null) { |
| for (const link of links) { |
| |
| link.classList.add('active') |
| let parentElement = link.parentElement |
| let count = 0 |
| const levelsLimit = 4 |
| |
| |
| |
| |
| |
| while (['UL', 'LI'].includes(parentElement.tagName) && count <= levelsLimit) { |
| if (parentElement.tagName === 'LI') { |
| |
| |
| |
| parentElement.querySelector('a:first-child').classList.add('active') |
| } |
| parentElement = parentElement.parentElement |
| count++ |
| } |
| } |
| } |
| })() |
| </script> |
| |
| </body> |
| </html> |
