binding-c/runtime/c/project/notes/array.txt - etch - Git at Google


 - - - - - - - - - - - - - - - - - -
 arrayvalue - nativearray discussion
 - - - - - - - - - - - - - - - - - -

 1. when reading arrays off the wire (tagdata input) the binding creates arrayvalue objects.

 2. question whether we can substitute our nativearray across the board.
    a. probably best not, since code up the line that is now mostly ported,
       would instead have to be rewritten for nativearray.

 3. should we however plan to always use 1-dim nativearray as the base of our arrayvalue,
    since if we need to be able to access elements, we can't simply reflect to an array as
    does java, we need the subscripting and primitive object instantiation facilities of the
    nativearray.

    a. however the arrayvalue is an arraylist under the covers. perhaps we should let the
       arrayvalue have either or both backing store types, arraylist or nativearray.
    b. that could create inconsistencies of access, in that accessing nativearray always
       returns a disposable, and accessing arrayvalue always returns a non-disposable.
    c. however we could well want to not store an object for every array element, a large
       byte array case in point.
    d. let's store only as arraylist for now, leaving the option open to store as
       nativearray if we need it.
    e. another possibility is to "lazy-load" the arraylist from the natarray.
       in this way, we always return a non-disposable from arrayvalue.get().
       when item[i] is requested, we look in the arraylist, lazy-allocating it if
       needed. if arraylist[i] is null, we populate it from natarray[i]. in either
       case we return the non-disposable arraylist[i].

 4. our arrayvalue has *both* an arraylist *and* a nativearray member.
    the natarray member currently is saved there when an arrayvalue was created *from* a nativearray.
    the natarray can be owned by the arrayvalue or not.
    a. arrayvalue has from() and to() nativearray methods
    b. arrayvalue_to_nativearray is limited to three dimensions of course.

 5. java code freely references the native array or the array value and expects both to be available.
    ArrayValue has getArray() which returns ArrayValue.array, the native array. TaggedData has
    Object fromArrayValue(ArrayValue) which returns the native array representation of the ArrayValue.
    So we really need to keep the 1-dim nativearray in the c arrayvalue. let's do it the lazy load way
    descibed above. the arrayvalue then handles the multi-dimensional aspect.


 - - - - - - - - - - - - - - - - - -
 serialization notes 1
 - - - - - - - - - - - - - - - - - -
 let's walk through writing some arrays.
 first, a 2-dimensional array.
 int x[2][3] = { {1,2,3}, {4,5,6} };

 0. we start with a nativearray, and call writeValue(ARRAY, nativearray);

 1. writeValue always puts the supplied byte typecode to the buffer first.
    so in this case, that is ARRAY. (1)

 2. next, switch(typecode) to ARRAY.
    a. convert the nativearray to an arrayvalue, passing ownership of na to av:
         etch_arrayvalue* av = new_arrayvalue_from
            (nativearray, typecode, customtype, size(?), 0, TRUE);
    b. call writeArray(av, vtor);

 3. writeArray calls startArray(arrayvalue);
    a. put the arrayvalue's content type byte. (2)
    b. if content type is custom, put array's custom struct type. (3)
    c. put intval arrayvalue's dimensions (4)
    d. put intval arrayvalue's itemcount  (5)

 4. writeArray calls writeValues(arrayvalue);
    a. elementVtor ev = av.eltvtor;
    b. foreach entry val in ev
       writeValue(ev, val);

 5. writeValue writes each value in the arrayvalue.
    this will be recursive for multidim arrays.
    a. in our case, the first arravalue has dim 2 and has 2 entries,
       which are the two 3-integer dimensions of x
       so writeValue(arravalue.value[i]); goes back to step 0.
    b. our inner recursion of writeValue (arravalue.value[i]) writes the int content (6)

 6. writeArray calls endArray(arrayvalue);
    a. endArray writes the EOD marker. (7)

 ARRAY INTEGER 2 2            (a1) (a2) (a4) (a5)
    ARRAY INTEGER 1 3    (a6):(b1) (b2) (b4) (b5)
    1                    (b6)
    2                    (b6)
    3                    (b6)
    EOD                  (b7)
    ARRAY INTEGER 1 3    (a6):(b1) (b2) (b4) (b5)
    4                    (b6)
    5                    (b6)
    6                    (b6)
    EOD                  (b7)
 EOD                     (a7)

 - - - - - - - - - - - - - - - - - -
 deserialization notes 1
 - - - - - - - - - - - - - - - - - -
 looking at the serialization, it seems as if we should do things a bit differently from the java version.
 when java creates an arrayvalue in which to read an array off the wire, it first allocates a java native
 array, and then wraps an arrayvalue around that. with the scanty information available, and without lookahead,
 we can't allocate our version of a nativearray, since (a) ours is top down, i.e. we need the entire byte blob
 first, and (b) we don't know the item count up front (we know the itemcount for the serialized *array*, in
 our example above that would be 2, i.e. 2 arrays of type and length not yet read. all we know at the time
 of array container allocation is the number of dimensions, and the actual serialized item count for that
 dimension.

 if it turns out we need data not yet read, we could conceivably look ahead for it. however the better approach
 might be to read into arrayvalues, and when done, construct the nativearray. lets explore that avenue here.

 0. we start with a serialized arrayvalue of dim 2, type int, whose members are 2 arrayvalues of dim 1.

 1. readValue finds ARRAY (a1)
    a. calls readArray()

 2. readArray
    a. calls startArray() getting a new arrayvalue back.

 3. startArray()
    a. reads the content type byte = INTEGER (a2)
    b. reads the dimensions count = 2  (a4)
    c. reads the item count = 2 (a5)
    d. allocates an arrayvalue (java code is lost)
    e. returns the new arrayvalue.

 4. readArray receives arrayvalue from startArray.
    readArray calls readValues(arrayvalue, vtor)

 5. readValues gets the element vtor for this dimension count
    value = readValue(arrayvalue); (a6)

    until value is EOD
       arrayvalue.add(value)
       value = readValue() ;

 6. readValue
    a. ARRAY (b1)

 7. readArray()
    arrayvalb = startArray();

 8. startArray()
    a. reads the content type byte INTEGER (b2)
    b. reads the dimensions count = 1 (b4)
    c. reads the item count = 3 (b5)
    d. allocates an arrayvalue (java code is lost)
    e. returns the new arrayvalb.


 9. readArray()
    a. readValues(arrayvalb);

 10. readValues gets the element vtor for this dimension count
     value = readValue(arrayvalb) = tiny 1; (b6)
     arrayvalb.add(value)
     value = readValue(arrayvalb) = tiny 2; (b6)
     arrayvalb.add(value)
     value = readValue(arrayvalb) = tiny 3; (b6)
     arrayvalb.add(value)
     value = readValue(arrayvalb) = EOD (b7)
     return;

 10. readArray() calls endArray which is a nop
     and returns arrayvalb.

 11. readValues (a) gets back Object arrayvalb and puts it to arrayvalue[0]
     (repeat 5-10)
      readValues (a) gets back Object arrayvalc and puts it to arrayvalue[1]


 999. readValue processing ARRAY receives arrayvalue result of readArray(),
    presumably an arrayvalue with 2 members, each member being an array dim1 with 3 members.
    and converts the arrayValue to nativearray.

	- - - - - - - - - - - - - - - - - -
	arrayvalue - nativearray discussion
	- - - - - - - - - - - - - - - - - -

	1. when reading arrays off the wire (tagdata input) the binding creates arrayvalue objects.

	2. question whether we can substitute our nativearray across the board.
	a. probably best not, since code up the line that is now mostly ported,
	would instead have to be rewritten for nativearray.

	3. should we however plan to always use 1-dim nativearray as the base of our arrayvalue,
	since if we need to be able to access elements, we can't simply reflect to an array as
	does java, we need the subscripting and primitive object instantiation facilities of the
	nativearray.

	a. however the arrayvalue is an arraylist under the covers. perhaps we should let the
	arrayvalue have either or both backing store types, arraylist or nativearray.
	b. that could create inconsistencies of access, in that accessing nativearray always
	returns a disposable, and accessing arrayvalue always returns a non-disposable.
	c. however we could well want to not store an object for every array element, a large
	byte array case in point.
	d. let's store only as arraylist for now, leaving the option open to store as
	nativearray if we need it.
	e. another possibility is to "lazy-load" the arraylist from the natarray.
	in this way, we always return a non-disposable from arrayvalue.get().
	when item[i] is requested, we look in the arraylist, lazy-allocating it if
	needed. if arraylist[i] is null, we populate it from natarray[i]. in either
	case we return the non-disposable arraylist[i].

	4. our arrayvalue has both an arraylist and a nativearray member.
	the natarray member currently is saved there when an arrayvalue was created from a nativearray.
	the natarray can be owned by the arrayvalue or not.
	a. arrayvalue has from() and to() nativearray methods
	b. arrayvalue_to_nativearray is limited to three dimensions of course.

	5. java code freely references the native array or the array value and expects both to be available.
	ArrayValue has getArray() which returns ArrayValue.array, the native array. TaggedData has
	Object fromArrayValue(ArrayValue) which returns the native array representation of the ArrayValue.
	So we really need to keep the 1-dim nativearray in the c arrayvalue. let's do it the lazy load way
	descibed above. the arrayvalue then handles the multi-dimensional aspect.


	- - - - - - - - - - - - - - - - - -
	serialization notes 1
	- - - - - - - - - - - - - - - - - -
	let's walk through writing some arrays.
	first, a 2-dimensional array.
	int x[2][3] = { {1,2,3}, {4,5,6} };

	0. we start with a nativearray, and call writeValue(ARRAY, nativearray);

	1. writeValue always puts the supplied byte typecode to the buffer first.
	so in this case, that is ARRAY. (1)

	2. next, switch(typecode) to ARRAY.
	a. convert the nativearray to an arrayvalue, passing ownership of na to av:
	etch_arrayvalue* av = new_arrayvalue_from
	(nativearray, typecode, customtype, size(?), 0, TRUE);
	b. call writeArray(av, vtor);

	3. writeArray calls startArray(arrayvalue);
	a. put the arrayvalue's content type byte. (2)
	b. if content type is custom, put array's custom struct type. (3)
	c. put intval arrayvalue's dimensions (4)
	d. put intval arrayvalue's itemcount (5)

	4. writeArray calls writeValues(arrayvalue);
	a. elementVtor ev = av.eltvtor;
	b. foreach entry val in ev
	writeValue(ev, val);

	5. writeValue writes each value in the arrayvalue.
	this will be recursive for multidim arrays.
	a. in our case, the first arravalue has dim 2 and has 2 entries,
	which are the two 3-integer dimensions of x
	so writeValue(arravalue.value[i]); goes back to step 0.
	b. our inner recursion of writeValue (arravalue.value[i]) writes the int content (6)

	6. writeArray calls endArray(arrayvalue);
	a. endArray writes the EOD marker. (7)

	ARRAY INTEGER 2 2 (a1) (a2) (a4) (a5)
	ARRAY INTEGER 1 3 (a6):(b1) (b2) (b4) (b5)
	1 (b6)
	2 (b6)
	3 (b6)
	EOD (b7)
	ARRAY INTEGER 1 3 (a6):(b1) (b2) (b4) (b5)
	4 (b6)
	5 (b6)
	6 (b6)
	EOD (b7)
	EOD (a7)

	- - - - - - - - - - - - - - - - - -
	deserialization notes 1
	- - - - - - - - - - - - - - - - - -
	looking at the serialization, it seems as if we should do things a bit differently from the java version.
	when java creates an arrayvalue in which to read an array off the wire, it first allocates a java native
	array, and then wraps an arrayvalue around that. with the scanty information available, and without lookahead,
	we can't allocate our version of a nativearray, since (a) ours is top down, i.e. we need the entire byte blob
	first, and (b) we don't know the item count up front (we know the itemcount for the serialized array, in
	our example above that would be 2, i.e. 2 arrays of type and length not yet read. all we know at the time
	of array container allocation is the number of dimensions, and the actual serialized item count for that
	dimension.

	if it turns out we need data not yet read, we could conceivably look ahead for it. however the better approach
	might be to read into arrayvalues, and when done, construct the nativearray. lets explore that avenue here.

	0. we start with a serialized arrayvalue of dim 2, type int, whose members are 2 arrayvalues of dim 1.

	1. readValue finds ARRAY (a1)
	a. calls readArray()

	2. readArray
	a. calls startArray() getting a new arrayvalue back.

	3. startArray()
	a. reads the content type byte = INTEGER (a2)
	b. reads the dimensions count = 2 (a4)
	c. reads the item count = 2 (a5)
	d. allocates an arrayvalue (java code is lost)
	e. returns the new arrayvalue.

	4. readArray receives arrayvalue from startArray.
	readArray calls readValues(arrayvalue, vtor)

	5. readValues gets the element vtor for this dimension count
	value = readValue(arrayvalue); (a6)

	until value is EOD
	arrayvalue.add(value)
	value = readValue() ;

	6. readValue
	a. ARRAY (b1)

	7. readArray()
	arrayvalb = startArray();

	8. startArray()
	a. reads the content type byte INTEGER (b2)
	b. reads the dimensions count = 1 (b4)
	c. reads the item count = 3 (b5)
	d. allocates an arrayvalue (java code is lost)
	e. returns the new arrayvalb.


	9. readArray()
	a. readValues(arrayvalb);

	10. readValues gets the element vtor for this dimension count
	value = readValue(arrayvalb) = tiny 1; (b6)
	arrayvalb.add(value)
	value = readValue(arrayvalb) = tiny 2; (b6)
	arrayvalb.add(value)
	value = readValue(arrayvalb) = tiny 3; (b6)
	arrayvalb.add(value)
	value = readValue(arrayvalb) = EOD (b7)
	return;

	10. readArray() calls endArray which is a nop
	and returns arrayvalb.

	11. readValues (a) gets back Object arrayvalb and puts it to arrayvalue[0]
	(repeat 5-10)
	readValues (a) gets back Object arrayvalc and puts it to arrayvalue[1]


	999. readValue processing ARRAY receives arrayvalue result of readArray(),
	presumably an arrayvalue with 2 members, each member being an array dim1 with 3 members.
	and converts the arrayValue to nativearray.