Google Git
Sign in
apache / mxnet-site / d5c5a235eef72f64ccaf224b834a20ef3287469b / . / versions / master / api / cpp / docs / api / classdmlc_1_1InputSplitShuffle.html
blob: 749f7e3b4d4443261cffb8af39d6c4b362c7b173 [file] [log] [blame]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.17"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>mxnet: dmlc::InputSplitShuffle Class Reference</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectalign" style="padding-left: 0.5em;">
<div id="projectname">mxnet
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.17 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
var searchBox = new SearchBox("searchBox", "search",false,'Search');
/* @license-end */
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
$(function() {
initMenu('',true,false,'search.php','Search');
$(document).ready(function() { init_search(); });
});
/* @license-end */</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>
<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0"
name="MSearchResults" id="MSearchResults">
</iframe>
</div>
<div id="nav-path" class="navpath">
<ul>
<li class="navelem"><a class="el" href="namespacedmlc.html">dmlc</a></li><li class="navelem"><a class="el" href="classdmlc_1_1InputSplitShuffle.html">InputSplitShuffle</a></li> </ul>
</div>
</div><!-- top -->
<div class="header">
<div class="summary">
<a href="#pub-methods">Public Member Functions</a> &#124;
<a href="#pub-static-methods">Static Public Member Functions</a> &#124;
<a href="classdmlc_1_1InputSplitShuffle-members.html">List of all members</a> </div>
<div class="headertitle">
<div class="title">dmlc::InputSplitShuffle Class Reference</div> </div>
</div><!--header-->
<div class="contents">
<p>class to construct input split with global shuffling
<a href="classdmlc_1_1InputSplitShuffle.html#details">More...</a></p>
<p><code>#include &lt;<a class="el" href="input__split__shuffle_8h_source.html">input_split_shuffle.h</a>&gt;</code></p>
<div class="dynheader">
Inheritance diagram for dmlc::InputSplitShuffle:</div>
<div class="dyncontent">
<div class="center"><img src="classdmlc_1_1InputSplitShuffle__inherit__graph.png" border="0" usemap="#dmlc_1_1InputSplitShuffle_inherit__map" alt="Inheritance graph"/></div>
<!-- MAP 0 -->
</div>
<div class="dynheader">
Collaboration diagram for dmlc::InputSplitShuffle:</div>
<div class="dyncontent">
<div class="center"><img src="classdmlc_1_1InputSplitShuffle__coll__graph.png" border="0" usemap="#dmlc_1_1InputSplitShuffle_coll__map" alt="Collaboration graph"/></div>
<!-- MAP 1 -->
</div>
<table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a name="pub-methods"></a>
Public Member Functions</h2></td></tr>
<tr class="memitem:afbce8f23fe731fb5ae087020f6274ab4"><td class="memItemLeft" align="right" valign="top">virtual&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#afbce8f23fe731fb5ae087020f6274ab4">~InputSplitShuffle</a> (void)</td></tr>
<tr class="separator:afbce8f23fe731fb5ae087020f6274ab4"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:af3f6ae9b61f4d18494d4bfbda0fab9c8"><td class="memItemLeft" align="right" valign="top">virtual void&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#af3f6ae9b61f4d18494d4bfbda0fab9c8">BeforeFirst</a> (void)</td></tr>
<tr class="memdesc:af3f6ae9b61f4d18494d4bfbda0fab9c8"><td class="mdescLeft">&#160;</td><td class="mdescRight">reset the position of <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> to beginning <a href="classdmlc_1_1InputSplitShuffle.html#af3f6ae9b61f4d18494d4bfbda0fab9c8">More...</a><br /></td></tr>
<tr class="separator:af3f6ae9b61f4d18494d4bfbda0fab9c8"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a491c4e5c44919e24d32403caeecdf468"><td class="memItemLeft" align="right" valign="top">virtual void&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#a491c4e5c44919e24d32403caeecdf468">HintChunkSize</a> (size_t chunk_size)</td></tr>
<tr class="memdesc:a491c4e5c44919e24d32403caeecdf468"><td class="mdescLeft">&#160;</td><td class="mdescRight">hint the inputsplit how large the chunk size it should return when implementing NextChunk this is a hint so may not be enforced, but <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> will try adjust its internal buffer size to the hinted value <a href="classdmlc_1_1InputSplitShuffle.html#a491c4e5c44919e24d32403caeecdf468">More...</a><br /></td></tr>
<tr class="separator:a491c4e5c44919e24d32403caeecdf468"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a768b224c5dad3d7cef5af3c5cbd08263"><td class="memItemLeft" align="right" valign="top">virtual size_t&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#a768b224c5dad3d7cef5af3c5cbd08263">GetTotalSize</a> (void)</td></tr>
<tr class="memdesc:a768b224c5dad3d7cef5af3c5cbd08263"><td class="mdescLeft">&#160;</td><td class="mdescRight">get the total size of the <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> <a href="classdmlc_1_1InputSplitShuffle.html#a768b224c5dad3d7cef5af3c5cbd08263">More...</a><br /></td></tr>
<tr class="separator:a768b224c5dad3d7cef5af3c5cbd08263"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:ab8812c94aa95744a475a1549b2b6f6a6"><td class="memItemLeft" align="right" valign="top">virtual bool&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#ab8812c94aa95744a475a1549b2b6f6a6">NextRecord</a> (<a class="el" href="structdmlc_1_1InputSplit_1_1Blob.html">Blob</a> *out_rec)</td></tr>
<tr class="memdesc:ab8812c94aa95744a475a1549b2b6f6a6"><td class="mdescLeft">&#160;</td><td class="mdescRight">get the next record, the returning value is valid until next call to NextRecord, NextChunk or NextBatch caller can modify the memory content of out_rec <a href="classdmlc_1_1InputSplitShuffle.html#ab8812c94aa95744a475a1549b2b6f6a6">More...</a><br /></td></tr>
<tr class="separator:ab8812c94aa95744a475a1549b2b6f6a6"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a88433138b96bc4c00d3e3bc8f2856025"><td class="memItemLeft" align="right" valign="top">virtual bool&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#a88433138b96bc4c00d3e3bc8f2856025">NextChunk</a> (<a class="el" href="structdmlc_1_1InputSplit_1_1Blob.html">Blob</a> *out_chunk)</td></tr>
<tr class="memdesc:a88433138b96bc4c00d3e3bc8f2856025"><td class="mdescLeft">&#160;</td><td class="mdescRight">get a chunk of memory that can contain multiple records, the caller needs to parse the content of the resulting chunk, for text file, out_chunk can contain data of multiple lines for recordio, out_chunk can contain multiple records(including headers) <a href="classdmlc_1_1InputSplitShuffle.html#a88433138b96bc4c00d3e3bc8f2856025">More...</a><br /></td></tr>
<tr class="separator:a88433138b96bc4c00d3e3bc8f2856025"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a2e3056434b2605d42f935b242d0e4a3d"><td class="memItemLeft" align="right" valign="top">virtual void&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#a2e3056434b2605d42f935b242d0e4a3d">ResetPartition</a> (unsigned rank, unsigned nsplit)</td></tr>
<tr class="memdesc:a2e3056434b2605d42f935b242d0e4a3d"><td class="mdescLeft">&#160;</td><td class="mdescRight">reset the Input split to a certain part id, The <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> will be pointed to the head of the new specified segment. This feature may not be supported by every implementation of <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a>. <a href="classdmlc_1_1InputSplitShuffle.html#a2e3056434b2605d42f935b242d0e4a3d">More...</a><br /></td></tr>
<tr class="separator:a2e3056434b2605d42f935b242d0e4a3d"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a036d26e7c95b13e65ff8accbf6fdb80e"><td class="memItemLeft" align="right" valign="top">&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#a036d26e7c95b13e65ff8accbf6fdb80e">InputSplitShuffle</a> (const char *uri, unsigned part_index, unsigned num_parts, const char *type, unsigned num_shuffle_parts, int shuffle_seed)</td></tr>
<tr class="memdesc:a036d26e7c95b13e65ff8accbf6fdb80e"><td class="mdescLeft">&#160;</td><td class="mdescRight">constructor <a href="classdmlc_1_1InputSplitShuffle.html#a036d26e7c95b13e65ff8accbf6fdb80e">More...</a><br /></td></tr>
<tr class="separator:a036d26e7c95b13e65ff8accbf6fdb80e"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="inherit_header pub_methods_classdmlc_1_1InputSplit"><td colspan="2" onclick="javascript:toggleInherit('pub_methods_classdmlc_1_1InputSplit')"><img src="closed.png" alt="-"/>&#160;Public Member Functions inherited from <a class="el" href="classdmlc_1_1InputSplit.html">dmlc::InputSplit</a></td></tr>
<tr class="memitem:a938da99f6cbaf1a4667bb00c75692e4b inherit pub_methods_classdmlc_1_1InputSplit"><td class="memItemLeft" align="right" valign="top">virtual bool&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplit.html#a938da99f6cbaf1a4667bb00c75692e4b">NextBatch</a> (<a class="el" href="structdmlc_1_1InputSplit_1_1Blob.html">Blob</a> *out_chunk, size_t n_records)</td></tr>
<tr class="memdesc:a938da99f6cbaf1a4667bb00c75692e4b inherit pub_methods_classdmlc_1_1InputSplit"><td class="mdescLeft">&#160;</td><td class="mdescRight">get a chunk of memory that can contain multiple records, with hint for how many records is needed, the caller needs to parse the content of the resulting chunk, for text file, out_chunk can contain data of multiple lines for recordio, out_chunk can contain multiple records(including headers) <a href="classdmlc_1_1InputSplit.html#a938da99f6cbaf1a4667bb00c75692e4b">More...</a><br /></td></tr>
<tr class="separator:a938da99f6cbaf1a4667bb00c75692e4b inherit pub_methods_classdmlc_1_1InputSplit"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a00a1f234a44de0950d1011495f41b265 inherit pub_methods_classdmlc_1_1InputSplit"><td class="memItemLeft" align="right" valign="top">virtual&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplit.html#a00a1f234a44de0950d1011495f41b265">~InputSplit</a> (void) <a class="el" href="3rdparty_2dmlc-core_2include_2dmlc_2base_8h.html#a80bde16113dc79e7adb7953e006d6b60">DMLC_THROW_EXCEPTION</a></td></tr>
<tr class="memdesc:a00a1f234a44de0950d1011495f41b265 inherit pub_methods_classdmlc_1_1InputSplit"><td class="mdescLeft">&#160;</td><td class="mdescRight">destructor <a href="classdmlc_1_1InputSplit.html#a00a1f234a44de0950d1011495f41b265">More...</a><br /></td></tr>
<tr class="separator:a00a1f234a44de0950d1011495f41b265 inherit pub_methods_classdmlc_1_1InputSplit"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table><table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a name="pub-static-methods"></a>
Static Public Member Functions</h2></td></tr>
<tr class="memitem:aa0277fb979b7466ba9afa7da86c0bfb6"><td class="memItemLeft" align="right" valign="top">static <a class="el" href="classdmlc_1_1InputSplit.html">InputSplit</a> *&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplitShuffle.html#aa0277fb979b7466ba9afa7da86c0bfb6">Create</a> (const char *uri, unsigned part_index, unsigned num_parts, const char *type, unsigned num_shuffle_parts, int shuffle_seed)</td></tr>
<tr class="memdesc:aa0277fb979b7466ba9afa7da86c0bfb6"><td class="mdescLeft">&#160;</td><td class="mdescRight">factory function: create input split with chunk shuffling given a uri <a href="classdmlc_1_1InputSplitShuffle.html#aa0277fb979b7466ba9afa7da86c0bfb6">More...</a><br /></td></tr>
<tr class="separator:aa0277fb979b7466ba9afa7da86c0bfb6"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="inherit_header pub_static_methods_classdmlc_1_1InputSplit"><td colspan="2" onclick="javascript:toggleInherit('pub_static_methods_classdmlc_1_1InputSplit')"><img src="closed.png" alt="-"/>&#160;Static Public Member Functions inherited from <a class="el" href="classdmlc_1_1InputSplit.html">dmlc::InputSplit</a></td></tr>
<tr class="memitem:a118dc34dbd121d5ff80cd904adee4c24 inherit pub_static_methods_classdmlc_1_1InputSplit"><td class="memItemLeft" align="right" valign="top">static <a class="el" href="classdmlc_1_1InputSplit.html">InputSplit</a> *&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplit.html#a118dc34dbd121d5ff80cd904adee4c24">Create</a> (const char *uri, unsigned part_index, unsigned num_parts, const char *type)</td></tr>
<tr class="memdesc:a118dc34dbd121d5ff80cd904adee4c24 inherit pub_static_methods_classdmlc_1_1InputSplit"><td class="mdescLeft">&#160;</td><td class="mdescRight">factory function: create input split given a uri <a href="classdmlc_1_1InputSplit.html#a118dc34dbd121d5ff80cd904adee4c24">More...</a><br /></td></tr>
<tr class="separator:a118dc34dbd121d5ff80cd904adee4c24 inherit pub_static_methods_classdmlc_1_1InputSplit"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a9b04228dac84268abf46719074700824 inherit pub_static_methods_classdmlc_1_1InputSplit"><td class="memItemLeft" align="right" valign="top">static <a class="el" href="classdmlc_1_1InputSplit.html">InputSplit</a> *&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="classdmlc_1_1InputSplit.html#a9b04228dac84268abf46719074700824">Create</a> (const char *uri, const char *index_uri, unsigned part_index, unsigned num_parts, const char *type, const bool shuffle=false, const int seed=0, const size_t batch_size=256, const bool recurse_directories=false)</td></tr>
<tr class="memdesc:a9b04228dac84268abf46719074700824 inherit pub_static_methods_classdmlc_1_1InputSplit"><td class="mdescLeft">&#160;</td><td class="mdescRight">factory function: create input split given a uri for input and index <a href="classdmlc_1_1InputSplit.html#a9b04228dac84268abf46719074700824">More...</a><br /></td></tr>
<tr class="separator:a9b04228dac84268abf46719074700824 inherit pub_static_methods_classdmlc_1_1InputSplit"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table>
<a name="details" id="details"></a><h2 class="groupheader">Detailed Description</h2>
<div class="textblock"><p>class to construct input split with global shuffling </p>
</div><h2 class="groupheader">Constructor &amp; Destructor Documentation</h2>
<a id="afbce8f23fe731fb5ae087020f6274ab4"></a>
<h2 class="memtitle"><span class="permalink"><a href="#afbce8f23fe731fb5ae087020f6274ab4">&#9670;&nbsp;</a></span>~InputSplitShuffle()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual dmlc::InputSplitShuffle::~InputSplitShuffle </td>
<td>(</td>
<td class="paramtype">void&#160;</td>
<td class="paramname"></td><td>)</td>
<td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
</div>
</div>
<a id="a036d26e7c95b13e65ff8accbf6fdb80e"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a036d26e7c95b13e65ff8accbf6fdb80e">&#9670;&nbsp;</a></span>InputSplitShuffle()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">dmlc::InputSplitShuffle::InputSplitShuffle </td>
<td>(</td>
<td class="paramtype">const char *&#160;</td>
<td class="paramname"><em>uri</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>part_index</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>num_parts</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">const char *&#160;</td>
<td class="paramname"><em>type</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>num_shuffle_parts</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">int&#160;</td>
<td class="paramname"><em>shuffle_seed</em>&#160;</td>
</tr>
<tr>
<td></td>
<td>)</td>
<td></td><td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>constructor </p>
<dl class="params"><dt>Parameters</dt><dd>
<table class="params">
<tr><td class="paramname">uri</td><td>the uri of the input, can contain hdfs prefix </td></tr>
<tr><td class="paramname">part_index</td><td>the part id of current input </td></tr>
<tr><td class="paramname">num_parts</td><td>total number of splits </td></tr>
<tr><td class="paramname">type</td><td>type of record List of possible types: "text", "recordio"<ul>
<li>"text": text file, each line is treated as a record input split will split on '\n' or '\r'</li>
<li>"recordio": binary recordio file, see <a class="el" href="recordio_8h.html" title="recordio that is able to pack binary data into a splittable format, useful to exchange data in binary...">recordio.h</a> </li>
</ul>
</td></tr>
<tr><td class="paramname">num_shuffle_parts</td><td>number of shuffle chunks for each split </td></tr>
<tr><td class="paramname">shuffle_seed</td><td>shuffle seed for chunk shuffling </td></tr>
</table>
</dd>
</dl>
</div>
</div>
<h2 class="groupheader">Member Function Documentation</h2>
<a id="af3f6ae9b61f4d18494d4bfbda0fab9c8"></a>
<h2 class="memtitle"><span class="permalink"><a href="#af3f6ae9b61f4d18494d4bfbda0fab9c8">&#9670;&nbsp;</a></span>BeforeFirst()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual void dmlc::InputSplitShuffle::BeforeFirst </td>
<td>(</td>
<td class="paramtype">void&#160;</td>
<td class="paramname"></td><td>)</td>
<td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>reset the position of <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> to beginning </p>
<p>Implements <a class="el" href="classdmlc_1_1InputSplit.html#a2f111755a3434097bdd2a765982180a7">dmlc::InputSplit</a>.</p>
</div>
</div>
<a id="aa0277fb979b7466ba9afa7da86c0bfb6"></a>
<h2 class="memtitle"><span class="permalink"><a href="#aa0277fb979b7466ba9afa7da86c0bfb6">&#9670;&nbsp;</a></span>Create()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">static <a class="el" href="classdmlc_1_1InputSplit.html">InputSplit</a>* dmlc::InputSplitShuffle::Create </td>
<td>(</td>
<td class="paramtype">const char *&#160;</td>
<td class="paramname"><em>uri</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>part_index</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>num_parts</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">const char *&#160;</td>
<td class="paramname"><em>type</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>num_shuffle_parts</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">int&#160;</td>
<td class="paramname"><em>shuffle_seed</em>&#160;</td>
</tr>
<tr>
<td></td>
<td>)</td>
<td></td><td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">static</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>factory function: create input split with chunk shuffling given a uri </p>
<dl class="params"><dt>Parameters</dt><dd>
<table class="params">
<tr><td class="paramname">uri</td><td>the uri of the input, can contain hdfs prefix </td></tr>
<tr><td class="paramname">part_index</td><td>the part id of current input </td></tr>
<tr><td class="paramname">num_parts</td><td>total number of splits </td></tr>
<tr><td class="paramname">type</td><td>type of record List of possible types: "text", "recordio"<ul>
<li>"text": text file, each line is treated as a record input split will split on '\n' or '\r'</li>
<li>"recordio": binary recordio file, see <a class="el" href="recordio_8h.html" title="recordio that is able to pack binary data into a splittable format, useful to exchange data in binary...">recordio.h</a> </li>
</ul>
</td></tr>
<tr><td class="paramname">num_shuffle_parts</td><td>number of shuffle chunks for each split </td></tr>
<tr><td class="paramname">shuffle_seed</td><td>shuffle seed for chunk shuffling </td></tr>
</table>
</dd>
</dl>
<dl class="section return"><dt>Returns</dt><dd>a new input split </dd></dl>
<dl class="section see"><dt>See also</dt><dd>InputSplit::Type </dd></dl>
</div>
</div>
<a id="a768b224c5dad3d7cef5af3c5cbd08263"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a768b224c5dad3d7cef5af3c5cbd08263">&#9670;&nbsp;</a></span>GetTotalSize()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual size_t dmlc::InputSplitShuffle::GetTotalSize </td>
<td>(</td>
<td class="paramtype">void&#160;</td>
<td class="paramname"></td><td>)</td>
<td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>get the total size of the <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> </p>
<p>Implements <a class="el" href="classdmlc_1_1InputSplit.html#af2fb3e0f9acd6bae599a398eb0758044">dmlc::InputSplit</a>.</p>
</div>
</div>
<a id="a491c4e5c44919e24d32403caeecdf468"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a491c4e5c44919e24d32403caeecdf468">&#9670;&nbsp;</a></span>HintChunkSize()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual void dmlc::InputSplitShuffle::HintChunkSize </td>
<td>(</td>
<td class="paramtype">size_t&#160;</td>
<td class="paramname"><em>chunk_size</em></td><td>)</td>
<td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>hint the inputsplit how large the chunk size it should return when implementing NextChunk this is a hint so may not be enforced, but <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> will try adjust its internal buffer size to the hinted value </p>
<dl class="params"><dt>Parameters</dt><dd>
<table class="params">
<tr><td class="paramname">chunk_size</td><td>the chunk size </td></tr>
</table>
</dd>
</dl>
<p>Reimplemented from <a class="el" href="classdmlc_1_1InputSplit.html#a58c7f48bec6bd1a880e59c4bbfdf4417">dmlc::InputSplit</a>.</p>
</div>
</div>
<a id="a88433138b96bc4c00d3e3bc8f2856025"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a88433138b96bc4c00d3e3bc8f2856025">&#9670;&nbsp;</a></span>NextChunk()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual bool dmlc::InputSplitShuffle::NextChunk </td>
<td>(</td>
<td class="paramtype"><a class="el" href="structdmlc_1_1InputSplit_1_1Blob.html">Blob</a> *&#160;</td>
<td class="paramname"><em>out_chunk</em></td><td>)</td>
<td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>get a chunk of memory that can contain multiple records, the caller needs to parse the content of the resulting chunk, for text file, out_chunk can contain data of multiple lines for recordio, out_chunk can contain multiple records(including headers) </p>
<p>This function ensures there won't be partial record in the chunk caller can modify the memory content of out_chunk, the memory is valid until next call to NextRecord, NextChunk or NextBatch</p>
<p>Usually NextRecord is sufficient, NextChunk can be used by some multi-threaded parsers to parse the input content</p>
<dl class="params"><dt>Parameters</dt><dd>
<table class="params">
<tr><td class="paramname">out_chunk</td><td>used to store the result </td></tr>
</table>
</dd>
</dl>
<dl class="section return"><dt>Returns</dt><dd>true if we can successfully get next record false if we reached end of split </dd></dl>
<dl class="section see"><dt>See also</dt><dd><a class="el" href="classdmlc_1_1InputSplit.html#a118dc34dbd121d5ff80cd904adee4c24" title="factory function: create input split given a uri">InputSplit::Create</a> for definition of record </dd>
<dd>
<a class="el" href="classdmlc_1_1RecordIOChunkReader.html" title="reader of binary recordio from Blob returned by InputSplit This class divides the blob into several i...">RecordIOChunkReader</a> to parse recordio content from out_chunk </dd></dl>
<p>Implements <a class="el" href="classdmlc_1_1InputSplit.html#a2e835c07ce77d8d873592e006c242156">dmlc::InputSplit</a>.</p>
</div>
</div>
<a id="ab8812c94aa95744a475a1549b2b6f6a6"></a>
<h2 class="memtitle"><span class="permalink"><a href="#ab8812c94aa95744a475a1549b2b6f6a6">&#9670;&nbsp;</a></span>NextRecord()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual bool dmlc::InputSplitShuffle::NextRecord </td>
<td>(</td>
<td class="paramtype"><a class="el" href="structdmlc_1_1InputSplit_1_1Blob.html">Blob</a> *&#160;</td>
<td class="paramname"><em>out_rec</em></td><td>)</td>
<td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>get the next record, the returning value is valid until next call to NextRecord, NextChunk or NextBatch caller can modify the memory content of out_rec </p>
<p>For text, out_rec contains a single line For recordio, out_rec contains one record content(with header striped)</p>
<dl class="params"><dt>Parameters</dt><dd>
<table class="params">
<tr><td class="paramname">out_rec</td><td>used to store the result </td></tr>
</table>
</dd>
</dl>
<dl class="section return"><dt>Returns</dt><dd>true if we can successfully get next record false if we reached end of split </dd></dl>
<dl class="section see"><dt>See also</dt><dd><a class="el" href="classdmlc_1_1InputSplit.html#a118dc34dbd121d5ff80cd904adee4c24" title="factory function: create input split given a uri">InputSplit::Create</a> for definition of record </dd></dl>
<p>Implements <a class="el" href="classdmlc_1_1InputSplit.html#a0f124b34e62d9319dca39fa42e5a59f7">dmlc::InputSplit</a>.</p>
</div>
</div>
<a id="a2e3056434b2605d42f935b242d0e4a3d"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a2e3056434b2605d42f935b242d0e4a3d">&#9670;&nbsp;</a></span>ResetPartition()</h2>
<div class="memitem">
<div class="memproto">
<table class="mlabels">
<tr>
<td class="mlabels-left">
<table class="memname">
<tr>
<td class="memname">virtual void dmlc::InputSplitShuffle::ResetPartition </td>
<td>(</td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>part_index</em>, </td>
</tr>
<tr>
<td class="paramkey"></td>
<td></td>
<td class="paramtype">unsigned&#160;</td>
<td class="paramname"><em>num_parts</em>&#160;</td>
</tr>
<tr>
<td></td>
<td>)</td>
<td></td><td></td>
</tr>
</table>
</td>
<td class="mlabels-right">
<span class="mlabels"><span class="mlabel">inline</span><span class="mlabel">virtual</span></span> </td>
</tr>
</table>
</div><div class="memdoc">
<p>reset the Input split to a certain part id, The <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a> will be pointed to the head of the new specified segment. This feature may not be supported by every implementation of <a class="el" href="classdmlc_1_1InputSplit.html" title="input split creates that allows reading of records from split of data, independent part that covers a...">InputSplit</a>. </p>
<dl class="params"><dt>Parameters</dt><dd>
<table class="params">
<tr><td class="paramname">part_index</td><td>The part id of the new input. </td></tr>
<tr><td class="paramname">num_parts</td><td>The total number of parts. </td></tr>
</table>
</dd>
</dl>
<p>Implements <a class="el" href="classdmlc_1_1InputSplit.html#aaa17b8be6b3f1da35ced68379e3e8dbe">dmlc::InputSplit</a>.</p>
</div>
</div>
<hr/>The documentation for this class was generated from the following file:<ul>
<li>/work/mxnet/3rdparty/dmlc-core/include/dmlc/<a class="el" href="input__split__shuffle_8h_source.html">input_split_shuffle.h</a></li>
</ul>
</div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Thu Jan 5 2023 03:47:40 for mxnet by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/>
</a> 1.8.17
</small></address>
</body>
</html>