gpfdist

start

• gpfdist_init

  • create hash table for sessions
  • parse_command_line
  • init libevent handler
  • http_setup # start the http server
    • set socket | bind | listen
    • set event for accept
  • sit and wait for events

• do_accept # Callback when the listen socket is ready to accept connections

  • accept connection
  • setup_read # set up for callback when socket ready for reading the http request
    • set do_read_request as EV_READ callback

• do_read_request

  • request_parse_gp_headers -> get xid/cid/sn ...
  • set_transform # setup transform errorfile and fstream options we create error file for readable transform here
  • session_attach
    • apr_hash_get by tid:path
    • if not craeted we should create new session
      • sessions_cleanup to remove outdated sessions
      • setup fstream options
      • fstream_open # open fstream and subprocess
        • gfile_open
          • if transform exists
          • bind fd->transform i/o routines
          • subprocess_open we create error file for writable transform here
  • handle_post_request
    • if final request -> go to request_end directly
    • if data request -> read data from request and do fstream_write
    • if first request -> go to request_end
    • request_end
      • if goto done_processing_request -> we will send header in request_end
      • otherwise -> we send header before request_end
      • if errorfile exists, we read from it, and send error file by proto1; after that send_eof or send_errmsg (which only happened in read path)
      • do session_end or session_detach
        • if all request in this session is detached, we do session_free
          • in session_free, we will do fstream_close. For post request, we can always get error message from error file after that.
          • destroy session_pool
      • if session_free, we can get error message and bind it to request_t->ferror
      • if we handle post request, we send ferror as header to client.

protocol0 write

each completed write contains at least three requests, with different X-GP-SEQ and X-GP-DONE

For the first request, X-GP-SEQ should be 1. X-GP-DONE should not exist. For the second request to nth request, X-GP-SEQ should be 2 - N. X-GP-DONE should not exist. For the last final request, X-GP-DONE should be 1. And X-GP-SEQ should be N.

Multiple segments can send requests at the same time. If the tid and path are same, they should be attached to the same session. However, if one or more of the segments are hanged, the requests from those segments may not be received by the gpfdist server in time. Gpfdist server can deallocate the session, and create another session for the same tid:path again for the same write from segments not sending first requests in time.

error report for write transform

If the subprocess failed not quick enough and writing is short, even the subprocess shall fail, the writing will be success due to the apr_file_write mechanism. The root cause is that we set out in apr_procattr_io_set as APR_NO_PIPE. So the error file won't be written in time. However, after the session is freed, and the fstream is closed, we must have done the apr_proc_wait. So that we can get error message from error file for sure. If we find an error exists, we should send the error message back to client in request_end. Also, we should clean up the error files.

If writing is long enough, we can also get error message during write. After wrote is failed, we will check the error file and send the error file to client in header if error exists. The client won‘t send DONE requests anymore since the error has already happened. So the server won’t have a chance to clean the session. However, that will not be an issue, because we will finally do the session clean up job by sessions_cleanup if we find that the session is idled for a while.

You will find that the cleaning up of an error file will only be done in session_free. So in most cases, we will only get error message in the final request in one of the segments. Other cases are writing failures happened. However, we still clean up the error file in session_free in those cases.